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ABSTRACT 


Turkey  has  been  battling  with  terrorist  groups  since  the  1980s.  In  total,  more  than  35,000 
Turkish  people  have  been  killed  by  terrorists.  The  majority  of  the  terrorist  activities  take 
place  near  the  Turkish-Iraqi  border,  which  is  characterized  by  rough  terrain.  Problems, 
such  as  lengthy  distances,  often  prevent  aircraft  from  reaching  the  area  before  the 
terrorists  achieve  their  objectives.  Limited  fuel  capacity  and  challenging  geographical 
conditions  are  other  issues  that  must  be  overcome.  Because  of  their  technical  capabilities 
and  longer  flight  times,  Unmanned  Combat  Aerial  Vehicles  (UCAVs)  may  enhance 
Turkey’s  ability  to  counter  active  terrorists  in  that  region.  In  this  research.  Map  Aware 
Non-uniform  Automata  (MANA)  is  used  to  model  different  counterterrorism  scenarios 
taking  place  along  the  Turkish-Iraqi  border.  We  examine  the  potential  effectiveness  of 
using  UCAVs  in  conjunction  with  manned  aircraft  to  detect  and  eliminate  terrorists  trying 
to  cross  the  border  and  attack  Turkish  military  assets.  For  this  purpose,  we  analyze  the 
data  from  102,800  simulated  air-to-ground  attacks  using  data  analysis  techniques,  such  as 
comparison  and  regression  analysis.  The  analysis  shows  that  UCAVs,  with  their 
additional  sensors  on  the  border  and  being  able  to  rapidly  attack  identified  targets,  are 
very  efficient  in  quickly  countering  terrorists  and  preventing  them  from  attacking  military 
forces. 
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EXECUTIVE  SUMMARY 


Terrorism  has  become  a  significant  problem  and  threatens  many  countries.  Turkey  has 
been  dealing  with  active  terrorist  groups  since  the  1980s.  In  total,  excluding  those 
unrecorded,  more  than  35,000  Turkish  people  have  been  killed  by  terrorists.  The  Turkish- 
Iraqi  border,  which  is  the  focus  of  this  thesis,  is  characterized  by  rough  geographical 
conditions,  deep  valleys,  and  steep  mountains.  The  challenging  terrain  of  the  Turkish- 
Iraqi  border  allows  terrorist  groups  to  make  surprise  attacks  on  military  and  police 
facilities  along  the  border  and  then  disappear.  The  dominant  terrorist  group  in  the  region 
has  been  the  Kurdish  Workers’  Party  (PKK). 

This  thesis  focuses  on  the  methods  that  the  Turkish  Air  Force  uses  to  detect  and 
eliminate  terrorists.  Only  the  Turkish-Iraqi  border  is  taken  into  consideration  (rather  than 
the  whole  southeastern  border  of  Turkey).  The  Turkish  Air  Force  has  used  the  methods 
described  below  to  kill  terrorists  or  prevent  them  from  attacking  military  stations, 
government  agencies,  and  military  and  police  lodgings  along  the  Turkish-Iraqi  border. 
After  obtaining  information  and  assessing  it  carefully,  the  Air  Force  immediately  sends 
manned  aircraft  to  the  region  to  conduct  attack  missions.  Problems,  such  as  lengthy 
distances,  often  prevent  aircraft  from  reaching  the  area  before  the  terrorists  achieve  their 
objectives.  Limited  fuel  capacity  and  challenging  geographical  conditions  are  other  issues 
that  must  be  overcome. 

The  majority  of  the  problems  that  Turkey  has  to  overcome  are  related  to  limited 
time  and  challenging  terrain.  Using  unmanned  combat  aerial  vehicles  (UCAVs)  in 
southeastern  Turkey  may  help  solve  some  of  the  problems.  UCAVs  are  weaponized 
unmanned  aerial  vehicles  that  can  be  used  for  air-to-ground  attack  missions  against  active 
terrorists.  This  thesis  analyzes  the  use  of  unmanned  combat  aerial  vehicles  in  conjunction 
with  manned  aircraft  against  active  terrorists  in  rough  terrain.  For  this  purpose,  we 
modeled  the  entire  Turkish-Iraqi  border  in  Map  Aware  Non-Uniform  Automata 
(MANA),  an  agent-based,  time-stepped,  stochastic  mission-level  model.  We  examine  the 
influence  of  UCAVs  against  active  terrorists  by  looking  at  the  number  of  terrorists  killed, 
the  time  required  to  kill  terrorists,  and  the  probability  that  the  terrorists  reach  their 


objective.  We  also  vary  the  parameters  of  F-16s,  unmanned  aerial  vehicles  (UAVs),  and 
UCAVs,  such  as  sensor  detection  ranges,  hit  probabilities,  speed,  and  reaction  times,  to 
explore  the  most  effective  parameters  for  mission  success. 

We  built  our  scenarios  by  taking  historical  incidents  as  a  reference.  We  are  trying 
to  figure  out  how  effective  UCAVs  might  be  in  that  area  as  an  alternative  to  F-16s.  For 
this  purpose,  we  constructed  eight  different  air-to-ground  attack  plans  comprising 
combinations  of  F- 16s,  UAVs,  and  UCAVs.  We  are  going  to  call  them  different  scenarios 
throughout  the  thesis  since  each  of  them  needs  to  be  run  separately  in  MANA,  one 
baseline  and  seven  alternate  scenarios.  Our  scenarios  include  three  air-to-ground  strike 
assets,  three  UAVs  for  intelligence,  surveillance,  and  reconnaissance  (ISR)  missions,  and 
three  terrorist  groups  that  attempt  to  cross  the  border  and  make  their  way  to  attack  three 
blue  battalions.  The  scenarios  also  have  partisans,  who  are  the  supporters  of  terrorists  in 
the  cities,  and  a  dummy  agent,  who  is  used  to  terminate  the  runs.  The  battle  area  is 
separated  into  three  regions,  and  each  air-to-ground  strike  asset,  UAV,  and  terrorist  group 
is  assigned  to  a  specific  region.  If  a  UAV  detects  and  classifies  any  terrorist  activity  along 
the  border,  it  immediately  reports  it  to  the  strike  assets  that  are  responsible  for  that 
specific  region.  Each  UAV  and  strike  asset  conducts  operations  in  a  dedicated  region,  and 
they  do  not  have  any  activity  in  other  regions.  The  following  figure  depicts  the  general 
overlay  of  the  scenarios. 
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There  are  three  regions  along  the  border.  The  combination  of  the  air  assets  by  scenario  is 
shown  on  the  upper  right  table.  UAVs  are  used  in  all  scenarios. 
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We  explore  the  effects  of  28  different  factors  in  this  thesis.  Among  these  28 
factors,  20  are  controllable  factors  and  eight  are  uncontrollable  factors.  Controllable 
factors  include  the  parameters  of  F-16s,  UCAVs,  and  UAVs.  Uncontrollable  factors 
include  the  parameters  of  terrorists  and  partisans.  We  used  a  nearly  orthogonal  Latin 
hypercube  (NOLH)  design,  which  provides  an  efficient  design  while  covering  a  broad 
section  of  input  space  with  257  design  points.  The  NOLH  design  also  guarantees  that  the 
factors  are  not  confounded.  We  ran  50  replications  for  each  design  point  and  each 
scenario,  and  came  up  with  a  final  data  set  based  on  102,800  air-to-ground  attack 
missions.  Afterwards,  the  data  was  analyzed  using  data  analysis  techniques,  such  as 
comparison,  partition  trees,  and  regression  analysis.  The  following  list  summarizes  the 
results  of  our  analysis. 

•  The  analysis  shows  that  UCAVs,  with  their  additional  sensors  on  the  border  and 
being  able  to  rapidly  attack  identified  targets,  are  very  efficient  in  quickly 
countering  terrorists  and  preventing  them  from  attacking  military  forces. 

•  In  general,  F-16s,  with  more  powerful  weapons,  are  better  at  inflicting  red 
casualties,  but  UCAVs  kill  more  quickly  and  are  better  at  preventing  the  terrorists 
from  attacking  blue. 

•  More  specifically,  the  scenarios  where  we  use  two  F-16s  and  one  UCAV  are  more 
advantageous  for  killing  terrorists  than  the  scenarios  with  one  F-16  and  two 
UCAVs.  On  the  other  hand,  the  scenarios  with  two  UCAVs  are  preferable  in  terms 
of  the  time  to  complete  the  mission  and  preventing  terrorists  from  reaching  their 
goal. 

•  The  most  common  conclusion  about  the  factors  is  that  weapons-related  variables 
have  a  strong  influence  on  the  number  of  terrorists  killed  while  sensor-related 
factors  are  the  most  effective  on  the  time  to  counter  the  terrorists  and  minimizing 
the  probability  that  terrorists  attack  blue. 

•  The  weapon  range  of  an  F-16  and  a  UCAV  are  the  two  most  influential  factors  on 
the  number  of  terrorists  killed.  The  sensor  classification  range  of  the  UAVs  have  a 
strong  effect  on  the  number  of  terrorists  killed,  the  time  to  complete  the  mission, 
and  the  probability  that  the  terrorists  attack  reach  their  goal. 

•  The  hit  probability  of  an  F-16  and  a  UCAV  are  two  of  the  most  significant  factors 
in  all  of  the  regression  model  fits. 

•  The  range  at  which  the  terrorists  classify  blue  is  a  significant  factor  for  the  time  to 
complete  the  mission. 

•  Finally,  the  ranges  at  which  the  terrorists  and  the  partisans  detect  blue  have  a 
significant  influence  on  the  probability  that  the  terrorists  attack  blue. 
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I. 


INTRODUCTION 


“Peace  at  home,  peace  in  the  world.  ” 

-  Mustafa  Kemal  Ataturk 

Terrorism  has  become  a  significant  problem  and  threatens  many  countries.  Many 
countries  spend  significant  amounts  of  money  to  protect  their  citizens  from  terrorist 
activities.  The  government  of  the  United  States  spent  more  than  half  a  trillion  dollars  to 
protect  against  and  respond  to  terrorist  activities  between  September  11,  2001,  and 
2012  [1].  Turkey  has  been  dealing  with  active  terrorist  groups  even  longer,  since  the 
1980s  [2].  In  total,  excluding  those  unrecorded,  more  than  35,000  Turkish  people  have 
been  killed  by  terrorists  [3].  This  chapter  provides  information  about  Turkey’s 
geographical  characteristics  and  challenges  surrounding  its  southeastern  border.  It  also 
describes  major  terrorist  organizations  that  threaten  Turkish  citizens  and  conventional 
forces  in  that  area  and  provides  examples  of  the  most  bloody  terrorist  attacks  in  Turkey’s 
history.  This  chapter  also  provides  an  explanation  about  current  counterterrorism 
methods  and  problems  that  the  Turkish  Air  Force  encounters  and  describes  potential 
solutions  for  these  problems. 

A.  MOTIVATION 

1.  Geography  of  Turkey  and  Challenges 

The  lands  of  Turkey  are  located  where  Asia  meets  Europe.  The  land  boundaries  of 
Turkey  stretch  2,949  kilometers,  and  coastlines  stretch  8,333  kilometers.  The  country 
shares  its  borders  with  the  European  countries  Bulgaria  and  Greece  and  the  Asian 
countries  Georgia,  Armenia,  Azerbaijan,  Iran,  Iraq,  and  Syria  (Figure  1)  [4].  Because 
Turkey  holds  lands  that  border  both  southeastern  Europe  and  southwestern  Asia,  it  has 
been  a  crossroads  country  between  the  two  continents.  Because  of  the  strategic 
significance  of  Turkey’s  location,  legal  travelers  as  well  as  terrorist  groups  use  it  for 
passage  and  refuge.  A  majority  of  anti-Turkey  terrorist  activities  take  place  in 

southeastern  Turkey.  Terrorist  groups  are  taking  advantage  of  political  and  economic 
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instability  and  regime  changes  in  neighboring  countries.  Since  the  beginning  of  Syria’s 
civil  war,  around  450,000  Syrian  refugees  have  crossed  the  Turkish-Syrian  border  and 
started  to  live  in  camps  thanks  to  Turkey’s  open-door  policy  [5],  The  Turkish- Iraqi 
border,  which  is  the  focus  of  this  thesis,  is  characterized  by  rough  geographical 
conditions,  deep  valleys,  and  steep  mountains.  The  challenging  terrain  of  the  Turkish- 
Iraqi  border  allows  terrorist  groups  to  make  surprise  attacks  on  military  and  police 
facilities  along  the  border  and  then  disappear. 

Turkey  has  also  been  used  as  a  corridor  by  terrorist  groups  for  smuggling.  One  of 
the  most  important  money  sources  of  terrorist  groups  is  drug  smuggling  [6].  Terrorists 
choose  Turkey  as  the  first  course  to  transport  illegal  materials  from  African  and  Asian 
countries  into  Europe.  Due  to  its  rugged  terrain,  terrorist  groups  opt  to  use  the  Turkish- 
Iraqi  border  as  a  passage  for  illicit  activities.  It  is  difficult  for  Turkish  conventional  forces 
to  prevent  those  terrorist  groups  from  committing  crimes  in  that  area  because  of  the  rough 
terrain. 
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2.  Terrorist  Organizations  in  Turkey 

Turkey  has  been  battling  terrorist  groups  for  a  long  time.  According  to  the  2014 
Global  Terrorism  Index  published  by  Institute  of  Economics  and  Peace,  Turkey  is  one  of 
the  countries  from  the  Organisation  for  Economic  Co-operation  and  Development 
(OECD)  that  experienced  the  highest  number  of  deaths  as  a  result  of  terrorist  attacks  [7]. 
Some  of  the  most  effective  terrorist  organizations  that  threaten  Turkey’s  southeastern 
borders  are  the  following. 

a.  PKK 

The  dominant  terrorist  group  in  the  region  has  been  the  Kurdish  Workers’  Party 
(PKK).  The  PKK  was  founded  by  Abdullah  Ocalan  on  November  27,  1978.  Their  main 
objective  is  to  establish  an  independent  Kurdish  state  within  Turkey’s  borders  through 
communist  revolution  and  armed  struggle  [8].  Although  the  PKK’s  leader  Abdullah 
Ocalan  was  captured  in  1999  and  sentenced  to  life  imprisonment  after  trial,  the  PKK 
never  ended  its  terrorist  activities  in  Turkey  and  northern  Iraq.  After  they  lost  their  leader, 
the  PKK  started  to  claim  that  it  changed  its  strategy  to  peaceful  ways.  In  order  to  prove 
itself  as  a  legitimate  political  organization  to  international  countries,  the  PKK  changed  its 
name  to  Kurdistan  Freedom  and  Democracy  Congress  (KADEK)  in  April  2002.  In  2003, 
the  terrorist  organization  changed  its  name  again  to  Kurdistan  Peoples’  Congress 
(KONGRA-GEL).  Despite  two  name  changes  and  a  declaration  of  policy  change,  the 
leading  members  of  the  PKK  did  not  change.  The  organization  is  still  led  by  Abdullah 
Ocalan,  who  gives  directions  from  prison  transmitted  to  the  other  members  of  the 
organization  by  his  lawyers.  Also,  other  leading  figures  such  as  Zubeyir  Aydar,  Murat 
Karayilan,  and  Cemil  Bayik  have  kept  their  leading  roles  in  the  PKK  [8].  They  have  also 
continued  terrorist  activities  in  Turkey.  The  PKK  has  also  been  on  the  U.S.  Department 
of  State’s  foreign  terrorist  organizations  list  since  1997  [9].  In  2010,  Europol,  a  European 
Union  Law  Enforcement  Agency,  described  the  PKK  as  an  ethno-nationalist  and 
separatist  terrorist  group  seeking  international  recognition  and  political  self-determination 
[10].  According  to  Turkey’s  Ministry  of  Foreign  Affairs,  legal  and  illegal  sources  of  PKK 
revenue  can  be  listed  as  follows: 


3 


•  Extortion 

•  Revenue  from  the  “special  nights”  organized  by  affiliates 

•  Sales  of  publications 

•  Revenue  from  commercial  establishments  belonging  to/affiliated  with  the 

organization 

•  Money  collected  through  drug  trafficking,  arms  smuggling,  and  human 

trafficking  [11], 

b.  ISIS 

Another  terrorist  organization  called  the  Islamic  State  of  Iraq  and  Syria-Sham 
(ISIS),  or  Islamic  State  of  Iraq  and  the  Levant  (ISIL),  recently  started  conducting  terrorist 
activities  along  the  Turkish-Iraqi  and  Turkish-Syrian  borders.  ISIS  was  preceded  by 
another  terrorist  group  known  as  the  Islamic  State  of  Iraq  (ISI),  which  was  formed  in 
2006.  ISI  was  comprised  of  different  rebel  groups,  most  significantly  Al-Qaeda.  The 
terrorist  organization  known  as  ISIS  was  established  in  April  2013  and  became  one  of  the 
main  terrorist  organizations  battling  against  conventional  government  forces  in  Iraq  and 
Syria  [12].  ISIS’s  primary  objective  has  been  to  establish  an  Islamic  state  ruled  by 
shariah  law  [13].  The  United  States  and  many  of  its  allies  have  been  fighting  ISIS  for  a 
couple  of  years,  but  in  February  2015,  President  Obama  formally  asked  Congress  for 
approval  of  military  force  against  ISIS  [14].  In  the  last  three  years,  ISIS  has  evolved  from 
a  classic  terrorist  organization  into  a  complicated,  well  organized  group  [12].  In  a  short 
period  of  time,  ISIS  has  replaced  the  PKK  as  the  most  deadly  terrorist  group  in  Turkey. 
Over  the  last  couple  of  years,  ISIS  has  been  responsible  for  a  quarter  of  all  the  deaths 
from  terrorist  attacks  in  Turkey  [7].  As  ISIS  continuously  increases  its  power,  it  becomes 
a  more  significant  threat  to  Turkey’s  borders. 

The  known  primary  funding  sources  of  ISIS  are  as  follows: 

•  Extortion 

•  Collection  of  taxes  and  fees  in  areas  under  its  control 

•  Selling  oil  from  fields  it  controls 

•  Looting  the  homes  of  people  who  flee  under  threat  of  the  militants  [13]. 
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c.  Al-Qaeda 

The  main  objective  of  Al-Qaeda  is  to  take  down  the  current  governments  in 
Muslim  countries  and  establish  a  new  Islamic  regime.  Al-Qaeda  calls  existing  Muslim 
countries  “puppets  of  the  West”  [16].  Al-Qaeda  has  been  active  in  many  countries  as  well 
as  along  Turkey’s  southeastern  border.  Al-Qaeda  usually  appears  with  its  suicide  attacks 
and  bombings.  The  main  sources  of  money  for  this  terrorist  group  are  donations  and  fees 
gathered  from  its  sympathizers  [15]. 

3.  Major  Terrorist  Incidents  in  the  Region 

Officially,  there  have  been,  more  than  35,000  Turkish  people  recorded  as  killed 
by  terrorists  since  the  1980s  [3].  Some  of  the  terrorist  attacks  have  occurred  in  big  cities, 
but  most  of  them  have  been  carried  out  in  rural  areas,  especially  in  southeastern  Turkey. 
There  are  hundreds  of  terrorist  attacks  committed  by  PKK,  ISIS,  Al-Qaeda,  and  others 
terrorist  organizations  in  Turkey.  Below  are  listed  some  of  the  most  violent  terrorist 
incidents,  defined  as  incidents  in  which  more  than  20  people  died. 

June  20,  1987:  Massacre  launched  in  Pinarcik  village  of  Omerli  district  in 
Mardin.  Sixteen  children,  six  women,  eight  men;  in  total  30  people  were 
killed  by  the  PKK  [16]. 

November  26,  1989:  In  Iki  Yaka  village  of  Yuksekova  district  in 
Diyarbakir,  21  people  were  killed,  and  nine  citizens  were  kidnapped  by 
the  PKK  [16]. 

June  11,  1990:  The  PKK  raided  Cevrimli  Village  of  Simak  and  killed  27 
people.  A  conflict  occurred  between  PKK  terrorists  and  the  village  guards, 
and  in  the  end  four  guards  lost  their  lives  [16]. 

October  1,  1992:  PKK  terrorists  raided  Cevizdali  village  of  Bitlis  and 
killed  30  people.  There  were  women  and  children  among  them,  and  25 
were  wounded.  Terrorists  set  the  village  on  fire  and  kidnapped  13  village 
guards  [16]. 

July  6,  1993:  The  PKK  made  an  armed  raid  on  Basbaglar  village  of 
Kemaliye  district  in  Erzincan  and  killed  28  people,  among  whom  there 
were  women,  and  wounded  three  people.  After  setting  57  houses  on  fire, 
the  terrorists  fled  away  [16]. 
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July  19, 1993:  The  PKK  attacked  Vanizer  village  of  Bahcesaray  district  in 
the  city  of  Van  and  killed  26  people,  22  of  them  were  women  [16], 

August  4,  1993:  PKK  terrorists  stopped  two  minibuses  traveling  between 
Kavakbasi  and  Yenidogan  villages  of  Mutki  district  in  Bitlis  and  shot  28 
people.  In  this  attack  15  people  were  killed  and  13  people  were  wounded 
[16]. 

October  5,  1993:  A  series  of  attacks  by  the  PKK  in  several  arable  fields 
and  villages  of  Batman,  Siirt,  and  Hakkari  caused  the  deaths  of  35  people 
(most  of  them  women  and  children).  Also,  10  people  were  on  the  victims 
list  as  wounded,  and  22  houses  were  sabotaged  [16]. 

October  25,  1993:  PKK  terrorists  raided  Erzurum’s  Cat  district  in  Yavi 
region  and  took  citizens  from  their  homes  and  gathered  them  together  in  a 
coffee  shop.  They  propagandized  their  declarations,  and  then  killed  them 
with  automatic  weapons.  During  the  attack,  35  people  died,  and  nearly  50 
were  wounded.  The  terrorists  then  set  the  houses  on  fire  and  escaped  [16]. 

November  15,  2003:  Thirty  people  were  killed  and  146  wounded  when 
car  bombs  shattered  two  synagogues  in  Istanbul.  Authorities  identified  two 
men  from  southeast  Turkey  as  the  suicide  bombers,  saying  the  attacks  bore 
the  hallmarks  of  the  al  Qaeda  network  [17]. 

May  11,  2013:  Reyhanli,  a  Turkish  town  on  the  border  with  Syria,  was 
attacked  by  Al-Qaeda  with  twin  car  bombs,  leaving  behind  an  official  toll 
of  52  deaths  and  146  injuries  [18]. 

B.  CURRENT  COUNTERTERRORISM  METHODS  AND  PROBLEMS 
1.  Methods 

This  thesis  focuses  on  the  methods  that  the  Turkish  Air  Force  uses  to  detect  and 
eliminate  terrorists.  It  does  not  look  at  other  methods  applied  by  Turkish  Army  assets  or 
police  forces.  Only  the  Turkish-Iraqi  border  is  taken  into  consideration  (rather  than  the 
whole  southeastern  border  of  Turkey).  The  Turkish  Air  Force  has  used  the  methods 
described  below  to  kill  terrorists  or  prevent  them  from  attacking  military  stations, 
government  agencies,  and  military  and  police  lodgings  along  the  Turkish-Iraqi  border. 

The  Turkish  Armed  Forces  have  a  couple  of  resources  that  provide  intelligence  to 

the  Turkish  Air  Force  in  southeastern  Turkey.  Unmanned  aerial  vehicles  (UAVs)  provide 

the  most  valuable  information.  UAVs  facilitate  intelligence,  surveillance,  and 

reconnaissance  (ISR)  missions  over  the  border  and  provide  information  about  terrorist 
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activities  and  locations.  Besides  UAVs,  intelligence  may  be  provided  by  another  source, 
such  as  Special  Operations  Forces  (SOF),  other  military  patrols,  or  police  forces. 
Sometimes,  villagers  provide  significant  information  about  terrorist  activities  in  the 
region  as  well. 

After  obtaining  information  and  assessing  it  carefully,  the  Air  Force  immediately 
sends  manned  aircraft  to  the  region  to  conduct  attack  missions.  Commonly  used  Air 
Force  bases  are  Diyarbakir  Air  Force  Base  (AFB),  located  in  Diyarbakir  city,  and  Erhac 
Air  Force  Base,  located  in  Malatya  city.  In  this  thesis,  we  assume  that  only  F-16s, 
deployed  in  Diyarbakir  AFB  are  available  for  this  kind  of  operation.  After  F-16s  take  off, 
they  deploy  to  the  area  and,  if  they  acquire  a  target,  drop  their  air  to  ground  weapons  on 
terrorist  militants.  F-16s  can  locate  and  attack  terrorists  by  using  their  onboard  sensors  or 
by  using  a  UAV’s  sensors.  However,  the  preferred  method  is  to  locate  the  terrorist  group 
with  onboard  sensors  based  on  coordinates  provided  by  UAVs  and  then  drop  a  bomb  and 
support  it  with  onboard  sensors.  After  releasing  their  bombs,  F-16s  may  continue  flying 
over  the  area  for  battle  damage  assessment  or  leave  the  area  and  return  to  base  and  rely 
on  UAVs  to  make  the  assessment. 

2.  Problems 

The  Turkish  Air  Force  has  been  encountering  many  problems  with  the  current 
counterterrorism  methods  in  southeastern  Turkey.  Diyarbakir  AFB  is  the  closest  Air 
Force  base  to  the  Turkish-Iraqi  border  that  accommodates  strike  aircraft.  The  distance 
between  Diyarbakir  AFB  and  the  most  western  part  of  the  Turkish-Iraqi  border  is 
approximately  115  nautical  miles  away,  which  takes  at  least  25  minutes  to  travel  for  an  F- 
16.  The  distance  between  Diyarbakir  AFB  and  the  most  eastern  part  of  the  Turkish-Iraqi 
border  is  approximately  220  nautical  miles  away,  which  takes  at  least  45  minutes  to 
travel  for  an  F-16.  We  also  need  to  add  5  to  15  minutes  to  the  travel  time  for  scramble 
response  delays.  Due  to  this  inevitable  delay,  sometimes  aircraft  cannot  reach  the  area  in 
time,  and  as  a  result,  they  cannot  prevent  terrorists  from  attacking  military  facilities  along 
the  border. 
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Terrorist  groups,  such  as  the  PKK,  ISIS,  and  Al-Qaeda,  which  operate  at  the 
Turkish-Iraqi  border,  have  supporters  in  urban  areas.  Sometimes,  terrorist  supporters  who 
live  in  the  cities  where  air  force  bases  are  located  inform  terrorists  about  aircraft  activities 
at  the  air  force  base.  As  a  result,  terrorists  sometimes  know  that  aircraft  are  coming  with 
fully  loaded  air-to-ground  weapons  and  hide  inside  caves  or  deep  valleys  before  the 
fighter  aircraft  reach  the  area.  UAVs  can  also  lose  contact  with  terrorists  and  strike 
aircraft  may  find  it  difficult  to  locate  and  drop  their  air-to-ground  weapons. 

Another  problem  the  Turkish  Air  Force  struggles  with  is  bad  weather  conditions. 
The  weather  at  the  border  area  may  be  clear,  but  if  weather  conditions  are  bad  in  the  city 
where  the  AFB  is  located,  the  strike  package  cannot  take  off  and  deploy  to  the  area. 

The  challenging  terrain  of  the  border  is  another  problem  for  strike  aircraft.  The 
topography  of  the  Turkish-Iraqi  border  is  rough,  which  provides  terrorists  a  strong 
advantage.  Also,  terrorists  are  well  adapted  to  the  area  because  they  have  been  operating 
in  the  region  for  decades.  Deep  valleys  and  caves  inside  the  mountains  are  perfect  places 
for  terrorists  to  hide  and  be  protected.  Therefore,  it  is  difficult  for  strike  aircraft  to 
conduct  attack  missions  in  such  terrain.  Figure  2  and  Figure  3  give  examples  of  the  rough 
terrain  and  terrorist  caves  near  the  Turkish-Iraqi  border. 

Another  significant  problem  is  that  manned  aircraft  have  a  limited  fuel  capacity 
relative  to  unmanned  aircraft.  The  operational  range  of  an  F-16  with  standard  air-to- 
ground  weapons  loaded  is  1,740  nautical  miles,  which  means  it  cannot  fly  over  the  area 
too  long  without  air-to-air  refueling  [19].  Air-to-air  refueling  also  requires  significant 
time,  depending  on  the  location  of  tanker  aircraft.  F-16s  may  need  to  return  to  base 
without  dropping  their  weapons  if  they  cannot  find  an  opportunity  to  attack  terrorists 
within  a  limited  time  interval. 
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Figure  2.  An  example  of  mountains  near  the  Turkish-Iraqi  border.  Turkish 
military  forces  are  conducting  an  operation  against  the  PKK.  Image 

from  etkihaber.com. 


Figure  3.  Turkish  military  forces  found  terrorist  caves  in  Turkish-Iraqi  border. 

Image  from  aktifhaber.com. 
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C.  SCOPE  OF  THE  THESIS 

The  majority  of  the  problems  that  Turkey  has  to  overcome  are  related  to  limited 
time  and  challenging  terrain.  Using  unmanned  combat  aerial  vehicles  (UCAVs)  in 
southeastern  Turkey  may  help  solve  some  of  the  problems.  UCAVs  are  weaponized 
UAVs  that  can  be  used  for  air-to-ground  attack  missions  against  active  terrorists.  More 
detailed  information  about  UAVs  and  UCAVs  is  given  in  Chapter  II. 

This  thesis  analyzes  the  use  of  UCAVs  in  conjunction  with  manned  aircraft 
against  active  terrorists  in  rough  terrain.  Replacing  manned  aircraft  and/or  F-16s  with 
UCAVs  may  help  secure  the  Turkish-Iraqi  border  more  effectively.  Because  UCAVs  can 
fly  for  more  than  24  hours,  they  can  apply  unlimited  pressure  on  terrorist  groups  and 
attack  them  right  after  detecting  and  classifying  them.  Using  UCAVs  in  some  regions  of 
the  border  and  F-16s  in  other  regions  might  also  increase  effectiveness.  For  this  purpose, 
we  modeled  the  entire  Turkish-Iraqi  border  in  Map  Aware  Non-Uniform  Automata 
(MANA),  an  agent-based,  time-stepped,  stochastic  mission-level  model.  More  detailed 
information  about  MANA  appears  in  Chapter  II.  We  examine  the  influence  of  UCAVs 
against  active  terrorists  by  looking  at  the  number  of  terrorists  killed,  the  time  required  to 
kill  terrorists,  and  the  probability  that  the  terrorists  reach  their  objective.  We  also  vary  the 
parameters  of  F-16s,  UAVs,  and  UCAVs,  such  as  sensor  detection  ranges,  hit 
probabilities,  speed,  and  reaction  times  to  explore  the  most  effective  parameters  for 
mission  success. 

Turkey  began  developing  its  own  unmanned  aerial  vehicle,  called  ANKA,  in  the 
2000s  to  reduce  its  reliance  on  foreign  industry.  ANKA  is  being  developed  by  Turkish 
Aerospace  Industries  (TAI).  It  will  be  delivered  to  the  Turkish  military  forces  during  the 
period  2016  to  2018  [20].  It  is  in  the  advanced  medium- altitude  long-endurance  (MALE) 
class  of  unmanned  aerial  systems.  ANKA  can  perform  day  and  night  and  in  all  weather 
conditions  for  more  than  24  hours  continuously  [21].  It  will  have  both  armed  and 
unarmed  versions.  Figure  4  depicts  unarmed  versions  of  ANKA,  which  will  be  used  for 
ISR  missions.  In  our  MANA  model,  we  use  ANKA’s  performance  characteristics  to 
simulate  those  of  the  UCAVs. 
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Figure  4.  ANKA  during  a  test  flight.  Image  from  tai.com.tr. 


D.  RESEARCH  QUESTIONS 

This  research  is  guided  by  the  following  questions: 

1.  How  might  UCAVs  enhance  Turkey’s  ability  to  secure  its  border 
characterized  by  rough  geographical  conditions? 

2.  What  combination  of  UCAVs  and  UAVs  provides  the  same  or  better 
effectiveness  as  the  combination  of  manned  aircraft  and  UAVs  currently 
in  use? 

3.  Is  there  a  combination  of  manned  aircraft,  UCAVs,  and  UAVs  that 
provides  the  same  or  better  effectiveness  than  the  manned  aircraft  and 
UAVs  in  use? 

4.  What  are  the  advantages  and  disadvantages  of  using  UCAVs  and  UAVs 
only,  manned  aircrafts  and  UAVs  only,  or  UCAVs,  manned  aircraft,  and 
UAVs? 

E.  THESIS  FLOW 

Chapter  II  contains  the  background  and  provides  useful  information  about 
vehicles,  tools,  techniques,  and  concepts.  We  provide  definitions  and  benefits  of  UAVs 
and  UCAVs,  discuss  agent-based  simulation  and  modeling,  and  provide  information 
about  the  Map-Aware  Non-Uniform  Automata  (MANA)  software  to  prepare  the  reader 
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for  the  following  chapters.  Chapter  Ill  covers  the  development  of  the  model  and  different 
scenarios  that  are  used  in  this  thesis.  A  detailed  description  of  the  agents  in  the  model  is 
explained  in  this  chapter  as  well.  Chapter  IV  explains  design  of  experiment  techniques 
and  simulation  runs.  First,  it  describes  the  factors  and  levels  that  we  vary  in  this  thesis. 
Then,  it  explains  the  nearly  orthogonal  Latin  hypercube  (NOLH)  design  that  was  used  to 
efficiently  explore  the  experiment  space.  Last,  it  covers  tools  and  techniques  that  we  used 
to  run  our  combat  simulation.  Chapter  V  concludes  the  thesis.  It  focuses  on  the  analysis 
of  the  output.  After  describing  the  analysis  tools  used,  it  covers  different  analysis 
techniques  and  the  results.  Finally,  it  provides  the  findings  and  gives  recommendations 
for  further  research. 
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II.  BACKGROUND  AND  LITERATURE  REVIEW 


“The  future  is  in  the  skies.  ” 

-  Mustafa  Kemal  Ataturk 

This  chapter  provides  a  review  of  the  tools,  systems,  and  eoncepts  that  we  use  in 
this  thesis.  First,  we  provide  definitions  and  background  about  UAVs,  with  an  emphasis 
on  UCAVs.  Seeond,  we  provide  definitions  and  a  discussion  of  the  benefits  of  agent- 
based  modeling  and  simulation  (ARMS).  We  also  provide  some  information  about  Map 
Aware  Non-Uniform  Automata  (MANA),  whieh  is  the  modeling  tool  used  in  this  thesis. 
Last,  we  provide  some  examples  of  related  studies  on  UAVs  and  UCAVs  for  border 
seeurity  and  air-to-ground  attaek  missions. 

A.  UNMANNED  VEHICLES 

1.  Definitions  of  UAV/  UCAV  and  Background 

Before  defining  UAVs  or  UCAVs,  we  need  to  define  an  unmanned  aircraft 
system  (UAS).  The  DOD  defines  a  UAS  as  “that  system  whose  components  inelude  the 
necessary  equipment,  network,  and  personnel  to  eontrol  an  unmanned  aireraft”  [22].  In 
other  words,  an  unmanned  aireraft  system  includes  an  unmanned  aerial  vehicle,  human 
element  (operator,  technician),  payload  (sensors,  weapons),  and  data  link,  among  other 
eomponents.  Figure  5  shows  the  major  components  of  a  UAS  [23].  A  UAV  is  only  one 
eomponent  of  an  unmanned  aireraft  system  (UAS).  Aeeording  to  the  DOD  UAS 
roadmap,  a  UAV  is  “a  powered,  aerial  vehicle  that  does  not  carry  a  human  operator,  uses 
aerodynamie  forces  to  provide  vehicle  lift,  ean  fly  autonomously  or  be  piloted  remotely” 
[24].  In  other  words,  a  UAV  is  an  aireraft  without  a  pilot  in  it. 
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Figure  5.  Elements  of  an  unmanned  aircraft  system  (UAS).  Image  from  [23]. 


UAVs  were  first  introduced  during  WWI  and  were  also  used  in  Vietnam,  but 
interest  in  them  increased  dramatically  during  the  Balkan  Peninsula  conflict  and 
Operation  Desert  Storm  [25].  Interest  in  UAVs  has  continued  to  increase  because  of  their 
great  contribution  to  mission  success  during  these  conflicts.  UAVs  are  critical 
components  of  ISR  missions  as  stated  in  the  DOD  UAS  roadmap. 

In  today’s  military,  unmanned  systems  are  highly  desired  by  combatant 
commanders  for  their  versatility  and  persistence.  By  performing  tasks  such 
as  surveillance;  signals  intelligence  (SIGINT);  precision  target 
designation;  mine  detection;  and  chemical,  biological,  radiological, 
nuclear  (CBRN)  reconnaissance,  unmanned  systems  have  made  key 
contributions  to  the  Global  War  on  Terror.  [26] 

UAVs  can  be  separated  into  two  distinct  groups:  remotely  piloted  UAVs  and 
autonomous  UAVs  [27].  Remotely  operated  UAVs  require  someone  to  control  the 
aircraft  from  a  distance  whereas  autonomous  UAVs  do  not  require  a  human  operator.  The 
primary  way  of  operating  UAVs  is  still  man-in-the-loop  control  rather  than  fully 
autonomous  control.  Militaries  use  UAVs  in  autonomous  mode  only  for  non-critical 
missions  [28].  This  is  because  of  the  fact  that  a  human  is  still  more  reliable  than  a 
computer  program  for  most  of  the  decision-making  process. 
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The  primary  application  of  UAVs  has  been  in  ISR,  but  in  the  last  several  years, 
governments  and  militaries  have  started  to  arm  their  UAVs.  A  weaponized  version  of  a 
UAV,  called  the  UCAV,  can  be  used  for  air-to-ground  or  air-to-air  attack  missions. 
UCAVs,  similar  to  manned  aircraft,  can  be  armed  with  a  variety  of  weapons,  such  as 
missiles  (AGM  114  Hellfire),  laser-guided  weapons  (GBU-12  Paveway  II),  and  GPS- 
guided  weapons  (GBU-38  JDAM).  They  can  be  used  to  attack  high-value,  fixed-ground 
targets  as  well  as  moving  ground  and  air  targets  in  military  operations. 

According  to  the  U.S.  DOD,  UAV  missions  may  include  but  are  not  limited  to: 

•  RSTA  missions 

•  Surveillance  for  search  and  rescue 

•  Deception  operations 

•  Maritime  operations  such  as  naval  surface  fire  support  (NSFS),  over-the-horizon 
targeting  (OTH-T),  ship  classification,  anti-ship  missile  defense  (ASMD), 
antisubmarine  warfare  (ASW) 

•  Electronic  warfare  (EW)  and  directed  energy  sensor  reconnaissance 

•  Nuclear,  biological,  and  chemical  (NBC)  reconnaissance 

•  Special  and  psychological  operations 

•  Meteorology  missions 

•  Route  and  landing  zone  reconnaissance  support 

•  Adjustment  of  indirect  fires  and  close  air  support  (CAS) 

•  Rear  area  security  support 

•  BDA 

•  Radio  and  data  relay  [29] 

Many  of  the  missions  listed  above  can  be  achieved  by  manned  aircraft  as  well,  but 
some  of  the  missions,  such  as  nuclear,  biological,  and  chemical  reconnaissance,  are  too 
risky  for  a  human.  Therefore,  UAVs  are  critical  components  for  militaries. 

2.  Benefits  of  Unmanned  Aircraft 

Nowadays,  governments  and  militaries  are  discussing  what  combination  of 
piloted  aircraft,  UAVs,  and  UCAVs  are  best  for  ISR  and  attack  missions.  According  to 
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the  DOD  UAS  roadmap,  “Unmanned  systems  have  proven  they  can  enhance  situational 
awareness,  reduce  human  workload,  improve  mission  performance,  and  minimize  overall 
risk  to  both  civilian  and  military  personnel,  and  all  at  a  reduced  cost”  [30],  In  other 
words,  UAVs  have  become  desirable  components  for  military  operations  as  well  as 
civilian  affairs  because  of  their  proven  benefits. 

Moreover,  military  services  can  purchase  and  operate  UAVs  for  less  money  than 
they  pay  for  manned  aircraft  [31],  Including  ground  control  stations,  the  procurement  cost 
of  a  UAV  is  40  to  80%  of  a  manned  aircraft  cost  and  operation — in  addition,  the 
maintenance  cost  of  a  UAV  is  20%  of  a  manned  aircraft  cost  [32],  According  to  the 
DOD,  “transportation  and  logistics  requirements  to  deploy  the  UAV  systems  are  usually 
smaller  than  other  airborne  intelligence  collection  resources.”  Also,  “extensive  special 
training  is  not  required  to  use  much  of  the  information  provided  by  UAVs.”  [33]  This  is 
because  UAVs  are  relatively  smaller  and  less  complex  systems  than  manned  aircraft. 

Among  all  benefits  listed  in  the  DOD’s  UAS  roadmap,  the  primary  benefit  of 
UAVs  is  their  ability  to  decrease  or  prevent  the  loss  of  personnel  life.  Even  when  the 
environmental  conditions  do  not  allow  using  manned  aircraft,  UAVs  can  still  provide 
effective  military  options.  For  example,  it  may  be  too  risky  to  assign  a  manned  aircraft  to 
an  NBC  reconnaissance  mission,  but  a  UAV  may  still  be  an  option  for  this  mission. 
Because  we  do  not  need  to  worry  about  a  pilot’s  life,  we  can  assign  UAVs  to  the  most 
dangerous  missions. 

UAVs  and  UCAVs  allow  military  forces  to  conduct  surveillance  and  attack 
missions  against  highly  defended  targets.  Because  UAVs  do  not  require  a  pilot  in  the 
cockpit,  they  can  withstand  extreme  conditions  that  their  piloted  counterparts  cannot, 
such  as  high  acceleration  forces  (g)  and  long  flight  times.  In  general,  removing  the  pilot 
from  the  aircraft  provides  the  ability  to  design  innovative  concepts  and  to  use  them  in  all 
environmental  conditions. 

B.  AGENT-BASED  MODELING  SIMULATION 

Significant  developments  in  computer  technology  and  the  high  operating  costs  of 
real  systems  have  given  rise  to  the  use  of  simulation  programs  in  military  services  [34]. 
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There  are  a  variety  of  military  applications,  such  as  training  methods,  attack/defense 
mission  rehearsals,  system  effectiveness,  new  concepts  of  operations,  and  military  tactics 
and  techniques  that  can  be  explored  or  analyzed  by  modeling  in  a  computer  environment. 
Furthermore,  due  to  the  lower  cost  of  computer  simulation  over  real  system  use,  military 
services  can  simulate  large-scale  military  operations  over  and  over  again  without 
spending  more  money.  Simulation  programs  can  also  be  used  to  bound  complex 
problems,  eliminate  irrelevant  options,  and  determine  the  areas  where  actual  tools  should 
be  used  or  field  experiments  are  needed. 

Agent-based  modeling  and  simulation  (ABMS)  is  one  of  the  modeling  and 
simulation  techniques  that  allow  users  to  represent  real-world  problems  within  a 
computer  program.  Sanchez  and  Lucas  describe  agent-based  simulations  as  “models 
where  multiple  entities  sense  and  stochastically  respond  to  conditions  in  their  local 
environments,  mimicking  complex  large-scale  system  behavior”  [35].  Agent-based 
modeling  allows  military  agencies  to  simulate  complicated  and  variable  interactions  of 
opposing  military  organizations  in  a  battlespace.  In  agent-based  modeling,  autonomous 
agents  such  as  aircraft,  tanks,  helicopters,  ships,  unmanned  vehicles  or  humans,  execute 
programmable  stochastic  behavioral  and  decision-making  rules.  As  a  result  of  individual 
agent  behaviors  and  interactions  between  different  agents  an  overall  simulation  result 
arises  as  a  sophisticated  global  model.  Castiglione  lists  ABMS’s  properties  as  follows: 

1 .  Agents  have  internal  states  which  can  be  represented  by  discrete  or 
continuous  variables. 

2.  Agents  may  change  their  states  after  interacting  with  another  agent  or  a 
state  change  may  cause  a  change  in  behavior. 

3.  Every  agent  may  interact  with  another  agent  locally  or  globally. 

4.  Rules,  which  range  from  simple  logic  to  complex  algorithms,  drive  agents’ 
behaviors. 

5.  The  system  evolves  over  the  time  and  both  time  and  space  can  be  discrete 
or  continuous  [36]. 

Agent-based  modeling  is  a  suitable  and  beneficial  tool  to  be  used  in  military 
applications.  In  his  paper,  Bonabeau  states  the  benefits  of  ABMS  techniques. 
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1 .  ABMS  captures  emergent  events  which  stem  from  interactions  of 
individual  entities. 

2.  In  many  cases,  ABMS  is  most  inherent  for  explaining  and  simulating  a 
system  formed  by  “behavioral”  entities. 

3.  ABMS  is  flexible  along  multiple  dimensions.  It  is  easy  to  modify  (add  or 
delete  agents)  an  agent-based  model  [37]. 

ABMS  can  be  executed  through  many  different  programs,  such  as  Microsoft 
Excel,  Java,  C++,  MATLAB,  and  R.  In  this  thesis,  we  use  an  ABMS  platform  called  Map 
Aware  Non-Uniform  Automata  (MANA)  to  explore  the  research  questions.  MANA  is 
one  of  the  most  user  friendly  modeling  tools,  and  for  that  reason,  it  has  been  widely  used 
by  NFS  faculty  and  students  [38]. 

C.  MANA 

This  section  draws  from  the  MANA  user  manual  versions  4  and  5.  For  more 
detailed  information  about  the  descriptions  and  features  of  the  modeling  tool,  the  reader 
should  look  to  the  MANA  manual. 

MANA  is  an  agent-based,  time-stepped,  stochastic,  mission-level  simulation 
environment.  It  was  developed  by  New  Zealand’s  Defense  Technology  Agency  (DTA) 
and  is  used  in  a  wide  range  of  national  and  international  scientific  defense  studies.  The 
development  of  MANA  has  been  ongoing  since  its  first  introduction  in  2000.  After  DTA 
started  to  increase  the  sophistication  of  its  agent-based  models  in  the  early  2000s,  it  has 
continually  improved  MANA  by  adding  new  features.  DTA  introduced  the  most  recent 
version  of  MANA,  MANA-V,  in  2009.  We  use  MANA-V  as  a  modeling  tool  in  this 
thesis.  Different  than  early  versions,  MANA-V  uses  a  vector-based  approach  for  agent 
movement  rather  than  a  grid-based  movement  scheme.  This  feature  of  MANA-V 
provides  a  number  of  advantages,  such  as  larger  battlefields  and  greater  flexibility  for 
developing  new  model  features.  Moreover,  MANA-V  provides  easier  interpretation  of 
scenarios  because  it  uses  international  system  (SI)  units  for  ranges.  Figure  6  illustrates  the 
two  different  movement  schemes  used  in  previous  versions  and  in  MANA-V  [39]. 
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Figure  6.  Two  distinct  movement  schemes  used  in  MANA:  (a)  grid-based 
approach  used  in  MANA  2,  3,  and  4;  (b)  vector-based  approach  used  in 
MANA-V.  Image  from  MANA  manual. 


Primary  elements  of  the  MANA  model  are  squads.  In  MANA,  a  squad  is  a 
collection  of  initially  homogeneous  agents  and  can  be  a  single  agent  or  a  group  of  agents. 
Each  squad  can  be  defined  as  friend,  enemy,  or  neutral.  This  affects  the  squad’s 
behaviors  when  interacting  with  other  squads.  Autonomous  agents  in  MANA  are 

•  Map-Aware:  Agents  are  aware  of  their  environment  through  their  organic  sensing 
capability  or  through  communication  with  other  agents. 

•  Non-uniform:  Each  major  agent  type  has  its  own  unique  set  of  properties. 

•  Automata;  Agents  react  independently  to  various  scenarios  based  on  the 
information  they  have  about  their  environment  [39]. 

Each  agent  in  MANA  has  personality  characteristics  that  can  drive  it  toward  or 
away  from  other  agents  or  objects  in  their  environment.  These  personality  settings  can  be 
changed  depending  on  user-defined  trigger  states  that  switch  the  personality  when  (or  if) 
certain  events  occur  (e.g.,  an  agent  being  shot  at).  A  state  is  a  collection  of  parameter 
values,  such  as  personality  weightings,  sensor  and  weapon  ranges,  concealment 
percentages,  and  speed  [39]. 

Movement  of  the  agents  is  limited  to  the  size  of  the  battlefield.  Direction  and 
speed  of  the  movement  is  determined  by  vector-based  algorithms  that  consist  of 
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personality  weightings  and  penalty  calculations  defined  by  the  user.  Terrain,  elevation, 
and  background  maps  can  be  loaded  into  the  model.  These  maps  are  basically  Windows 
bitmaps  that  the  user  can  modify  using  the  built  in  MANA  map  editor  or  using  some 
software  such  as  Microsoft  Paint  or  Adobe  Photoshop.  The  terrain  map  affects  agents’ 
“Going,”  “Cover,”  and  “Concealment.”  “Going”  refers  to  how  the  terrain  affects  an 
agent’s  speed.  “Cover”  defines  the  level  of  protection  of  an  agent  against  enemy  attacks. 
“Concealment”  refers  to  the  level  of  invisibility  of  an  agent  in  the  terrain.  “Going,” 
“Cover,”  and  “Concealmenf’  parameters  are  determined  by  the  colors  which  form  the 
whole  terrain  map.  The  elevation  map  is  gray-scale  map  ranging  from  black  to  white,  and 
it  is  used  for  line-of-sight  (LOS)  calculations.  The  elevation  map  provides  a  realistic 
implementation  of  the  battlefield.  For  example,  an  agent  on  one  side  of  a  hill  cannot  see 
or  shoot  at  an  agent  on  the  other  side  of  the  hill.  The  background  map  does  not  affect 
agents.  It  can  be  stored  to  enhance  a  scenario’s  appearance  to  the  user  [39]. 

An  agent  can  detect  and  classify  another  agent  within  a  range  defined  by  the 
developer  by  using  its  own  sensor  (or  sensors).  The  simple  mode  of  the  sensor  operates  as 
cookie-cutter  sensor  while  in  the  advanced  mode  a  user  can  define  different  sensor 
characteristics  for  different  ranges.  For  example,  it  is  more  likely  to  detect  nearby  agents 
than  farther  agents.  No  matter  if  the  simple  mode  or  the  advanced  mode  is  used,  detection 
requires  a  LOS  between  two  agents.  Similar  to  sensors,  an  agent  can  have  weapons  in  the 
form  of  a  kinetic  energy  weapon  and/or  a  high  explosive  weapon.  Agents  can  use  those 
weapons  against  enemy  agents.  MANA  uses  a  user-defined  range  probability  table  to 
determine  the  degree  of  effect  of  a  weapon  [39]. 

In  MANA,  agents  can  communicate  with  other  agents  in  the  same  squad  or  with 
agents  in  different  squads.  Communication  ranges,  latency,  reliability,  and  capacity  of  the 
transmission,  and  the  type  of  information  that  is  intended  to  be  passed,  is  determined  by 
the  user.  Communications  between  squads  can  be  visualized  in  a  situational  awareness 
(SA)  map.  There  are  two  types  of  SA  maps,  squad  SA  maps  and  inorganic  squad  SA 
maps.  Any  enemy  detection  made  by  one  agent  in  the  squad  is  put  onto  the  squad’s  SA 
map  so  that  other  agents  in  the  same  squad  can  access  the  information.  Similar  to  the 
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squad  SA  map,  the  inorganic  squad  SA  map  shows  detections  made  by  an  agent  from  a 
different  squad  that  been  communicated  to  the  squad  [39]. 

In  terms  of  agent-based  models,  MANA  has  a  number  of  advantages.  Those 
advantages  are  listed  in  the  MANA  manual  as  follows. 

•  It  is  user  friendly  and  has  an  easily  navigable  user  interface.  Scenarios  can  be 
quickly  edited  “on  the  fly”  during  development,  allowing  different  ideas  to  be 
quickly  explored.  In  this  regard,  MANA  can  be  thought  of  as  a  “sketch  pad”  for 
trying  new  military  tactics. 

•  MANA  is  supplied  as  a  pre-compiled  executable,  so  it  runs  relatively  fast 
compared  with  other  agent-based  models  built  on  interpreted  languages. 

•  The  model  has  a  built-in  data  farming  capability  which  allows  rapid  exploration  of 
parameter  spaces. 

•  MANA  can  model  communications  links  between  agents  for  information  sharing. 
Hence,  aspects  of  network-centric  warfare  (NCW)  can  be  studied. 

•  To  model  terrain  features,  MANA  uses  colored  bitmaps.  This  has  the  advantage 
that  terrain  features  can  be  quickly  altered  using  a  simple  image  editor  (for 
example,  MS  Paint),  while  a  scenario  is  in  development. 

•  MANA  has  a  large  number  of  trigger  states  that  agents  can  go  into  based  upon 
various  events  occurring  on  the  battlefield.  This  increases  the  richness  of  behaviors 
and  insights  that  can  be  obtained  from  scenarios  [39]. 

D.  LITERATURE  REVIEW 

UAVs  and  UCAVs  have  been  improving  every  day,  and  they  are  now  regarded  as 
a  main  element  of  modern  warfare.  Because  of  their  undebatable  contribution  to  battles, 
especially  in  asymmetric  warfare  environments,  new  and  improved  versions  of  UAVs 
and  UCAVs  are  being  produced  by  many  different  countries  [40].  A  significant  number 
of  studies  that  address  UCAVs  and  border  security  issues  depict  the  importance  of  the 
matter.  We  reviewed  some  of  these  studies  below. 

In  his  thesis,  Bessemer  [41]  points  out  the  United  States  Air  Force’s  demand  for 
UCAVs  and  seeks  a  quick  fix  to  satisfy  this  demand  until  fully  capable  unmanned 
vehicles  are  produced  [41].  He  argues  that  the  most  effective  way  to  decrease  risk-of-life 
and  budget  costs  is  to  introduce  an  F-16  unmanned  aerial  system  (UAS)  aircraft  for 
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combat.  He  illustrates  “how  an  effective  F-16  UAS  can  synchronize  recourses  to  properly 
complete  UCAV  development  while  instantly  redueing  risk  of  life”  [41],  His  overall 
assessment  is  that  the  F-16  UAS  is  a  good  solution  for  American  air  supremacy. 

In  his  study,  Kumar  [42]  expresses  that  major  improvements  in  UAVs  have  been 
in  the  role  of  tactical  reconnaissance  [42].  He  demonstrates  that  UAVs  are  perfeet  tools  to 
fill  the  information  gap  on  the  battlefield.  If  militaries  employ  UAVs  in  an  efficient 
manner,  the  inherent  eharaeteristies  of  UAVs  make  them  capable  of  complementing 
manned  aircraft  in  the  role  of  tactical  reconnaissance  [42].  Although  militaries  started  to 
use  UAVs  instead  of  manned  aircraft  for  some  missions,  they  always  seek  more  efficient 
designs  and  employment  methods  for  UAVs,  as  Kumar  pointed  out. 

In  his  thesis,  Beales  [43]  reviews  the  U.S.  government’s  decision  to  end  F-22 
produetion  and  switeh  to  UCAV  proeurement.  He  states  that  it  is  very  obvious  that 
UCAVs  are  cheaper  and  more  persistent,  but  on  the  other  hand,  terminating  F-22 
produetion  deereases  the  USAF’s  ability  to  defend  the  homeland  against  some  potential 
threats  [43].  As  a  result,  he  suggests  a  eoneentration  toward  UCAVs  in  future 
proeurement  while  improving  versatility  and  availability. 

In  his  thesis,  Gill  [44]  construets  a  simulation  model  of  a  strike  scenario  that 
foeuses  on  the  eoordination  of  the  Navy  Unmanned  Combat  Air  System  (NUCAS),  F/A 
18  Super  Hornet,  and  the  F-35C  Lightning  II.  By  using  design  of  experiment  teehniques 
and  making  12,000  simulation  runs,  he  evaluates  the  results  in  order  to  determine  the 
number  of  aireraft  needed  for  mission  sueeess  and  the  factors  that  are  required  to  reduee 
friendly  aireraft  losses  [44].  He  determines  that  a  four  NUCAS  division  is  advantageous 
and  a  stealth  requirement  is  erueial  in  future  aircraft  development. 

In  his  thesis,  Sulewski  points  out  that  “the  future  foree  will  rely  heavily  on  UAVs 
to  provide  eyes  on  the  battlefield”  [45].  He  analyzes  the  effeet  of  the  numbers  and 
capabilities  of  armed  and  unarmed  UAVs  in  the  U.S.  Army’s  future  combat  battalions. 
For  this  purpose,  Sulewski  builds  a  simulation  model  in  MANA  and  makes  46,440 
computational  experiments.  He  finds  that  UAVs  signifieantly  enhanee  the  performanee 
and  armed  UAVs  (UCAVs)  are  eritieal  in  the  opening  phase  of  the  battle. 
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In  his  study,  Hume  examines  “the  issue  of  weaponized  UAV  integration  into  the 
battlespace  from  the  standpoint  of  doctrine,  operational  concept,  and  roles  and  missions” 
[46],  Hume  makes  some  recommendations  about  how  best  to  employ  weaponized  UAVs 
in  the  future.  He  recommends  giving  weaponized  UAV  missions  to  the  U.S.  Air  Force,  to 
create  a  joint  acquisition  strategy,  to  establish  joint  standards  for  unmanned  aircraft  (UA) 
employment,  and  to  improve  command  and  control  systems  of  weaponized  UAs. 

In  her  thesis,  Ozcan  analyzes  the  effectiveness  of  the  UAVs  in  helping  secure  the 
Turkish-Iraqi  border  characterized  by  rough  terrain  and  active  terrorists  [47],  She 
modeled  the  Turkish-Iraqi  border  in  MANA  and  examined  the  potential  impact  of  UAVs 
on  detecting  and  classifying  terrorists  trying  to  pass  the  border  and  attack  the  blue  forces. 
The  results  from  the  103,200  runs  showed  that  UAVs  are  very  efficient  in  the  detection 
and  the  classification  of  terrorists  in  this  region.  She  also  points  out  that  the  most 
significant  factors  of  UAVs  are  the  detection  and  the  classification  performance. 

All  of  the  studies  and  theses  that  we  summarized  discuss  different  perspectives  on 
the  use  of  UCAVs.  Some  of  them,  such  as  [46],  point  out  the  strategic  importance  of 
UCAVs,  whereas  others,  such  as  [45]  and  [42],  discuss  tactical  level  contributions. 
Moreover,  some  researchers,  such  as  [44],  examined  the  results  of  using  UCAVs  and 
manned  aircraft  together  in  a  battlefield,  while  others,  such  as  [43],  discuss  using  only 
UCAVs  alone.  Building  on  the  literature,  this  thesis  examines  the  importance  of  using 
UCAVs  in  conjunction  with  manned  aircraft  in  areas  of  rough  terrain. 
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III.  MODEL  AND  SCENARIO  DESCRIPTION 


In  this  chapter,  we  explain  the  model  development  process  and  describe  the 
seenarios  we  constructed  to  examine  UCAV  effeetiveness.  First,  we  discuss  how  we 
designed  the  battlefield  and  constructed  the  maps  we  needed  for  the  model.  Next,  we 
deseribe  our  baseline  scenario  and  seven  alternative  scenarios  in  detail.  Finally,  we 
explain  the  agents  in  the  model. 

A.  MODEL  DEVELOPMENT 

Map  Aware  Non-Uniform  Automata  Veetor  (MANA-V)  was  used  as  a  modeling 
tool  in  this  thesis.  It  took  approximately  two  months  to  complete  the  model  and  make  it 
ready  to  run.  This  two-month  period  included  learning  MANA,  by  reading  the  manual, 
and  reviewing  old  models;  eonstrueting  maps,  elevation,  background,  and  terrain  maps; 
building  the  basic  scenario;  and  finally,  building  alternative  scenarios. 

1.  Battlefield 

For  this  thesis,  the  384  km  Turkish-Iraqi  border  was  modeled  as  a  battlefield  in 
MANA.  384  km  is  the  real  length  of  the  Turkish-Iraqi  border;  as  far  as  air  distanee  is 
eoncemed,  it  is  approximately  222  km.  This  served  as  the  length  of  the  battlefield.  108 
km  was  chosen  as  the  width  of  the  battlefield  to  inelude  blue  battalions  on  the  north  of 
the  border  and  to  adjust  the  map  ratio.  Therefore,  a  222  km  x  108  km  terrain  was 
eonsidered  the  area  of  interest.  The  battlefield  was  bounded  by  the  Turkish-Iranian  border 
on  the  east  and  the  Turkish-Syrian  border  on  the  west.  We  separated  the  battlefield  into 
three  independent  operational  regions.  Each  agent  operates  in  a  dedieated  region  and  does 
not  involve  any  activity  in  other  regions.  We  display  those  regions  on  the  map  in  the 
scenario  deseription  section  later  in  this  chapter. 

2.  Map  Construction 

As  explained  in  Chapter  II,  MANA  needs  elevation  and  terrain  map  input  if  the 
user  wants  topography  to  affeet  the  agents.  Also,  MANA  aeeommodates  a  background 
map  to  improve  visualization  of  the  seenario.  In  this  thesis,  all  three  types  of  maps  were 
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used  because  together  they  constructed  a  realistic  mission-level  model,  which  included 
humans  moving  on  the  ground,  stationary  ground  assets,  and  aerial  vehicles. 


a.  Elevation  and  Background  Maps 

The  elevation  map  in  gray-scale  helps  formulate  line-of-sight  (LOS)  calculations. 
After  selecting  the  area  of  interest,  an  elevation  map  was  constructed  using  a  variety  of 
sources  and  software.  Environmental  Systems  Research  Institute  (ESRI),  the  international 
developer  of  the  geographic  information  system  (GIS),  was  used  to  obtain  raw  elevation 
data  for  the  Turkish-Iraqi  border.  We  used  ArcGIS  software  online  to  create,  display,  and 
analyze  geospatial  data  developed  by  ESRI.  Elevation  data  was  stored  as  a  keyhole 
markup  language  zipped  (kmz)  extension  file  that  could  be  used  by  Google  Earth.  After 
opening  the  downloaded  data  in  Google  Earth,  we  saved  a  picture  of  the  area  of  interest 
in  a  picture  format.  Finally,  Microsoft  Paint  helped  us  adjust  the  size  and  resolution,  and 
store  the  final  version  in  bitmap  (bmp)  format,  which  is  an  acceptable  format  for  MANA 
(see  Figure  7).  In  this  study,  real  world  elevation  was  defined  between  650  and  3200 
meters  above  sea  level  for  the  lowest  and  highest  points,  respectively.  We  also  used  the 
elevation  map  as  a  background  map  in  the  model  by  adding  the  Turkish-Iraqi  border  on 
the  elevation  map  as  a  line.  The  background  map  does  not  affect  agents.  It  simply 
enhances  a  scenario’s  appearance  to  the  user. 


b.  Terrain  Map 

The  gray-scale  elevation  map  was  referenced  to  construct  an  accurate  terrain  map. 
The  lightest  parts  of  the  elevation  map  are  the  tops  of  the  mountains  while  the  darkest 
parts  are  valleys.  We  used  Microsoft  Visual  Studio  to  construct  the  terrain  map.  This 
process  took  more  than  a  week  because  Visual  Studio  separates  the  elevation  map  into  its 
pixels  and  the  user  needs  to  cross-reference  the  colors  of  every  pixel  with  the  elevation 
map.  Using  more  improved  software,  such  as  Photoshop,  is  highly  recommended  for 
terrain  map  construction  in  order  to  save  days  or  even  weeks. 

The  terrain  map  comprises  three  different  colors,  which  represent  three  different 
terrain  characteristics  for  mountains,  off-road,  and  valleys.  Each  color  on  the  terrain  map, 
or  in  other  words  each  terrain  type,  has  a  certain  “Going,”  “Cover,”  and  “Concealment” 
value  ranging  from  0.0  to  1.0.  A  value  of  0.0  means  “No”  “Going,”  “Cover”  or 
“Concealment,”  while  a  value  of  1.0  means  “Full”  “Going,”  “Cover”  or  “Concealment.” 
Terrain  types  and  parameters  are  depicted  in  Figure  8. 


Figure  8.  Terrain  types  and  parameters. 
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3.  Model  Time  Step 

MANA  is  a  time-stepped  model,  (i.e.,  all  events  occur  in  discrete  intervals).  In 
time-stepped  models,  each  time  the  main  script  is  executed,  the  simulation  time  is 
incremented  by  the  simulation  time  step.  In  other  words,  the  computer  makes  all  the 
required  calculations  for  every  time  step,  and  the  next  calculation  occurs  in  the  next  time 
step.  Using  large  time  steps  results  in  fast  but  inaccurate  simulations  because  the  user 
may  miss  some  important  events.  On  the  other  hand,  small  time  steps  lead  to  more 
precise  simulations,  but  will  take  more  time.  If  the  model  is  too  complex,  the  run  time 
will  be  even  more  because  the  computer  will  have  to  make  more  calculations  every  time 
step.  In  our  model,  the  time  step  was  set  to  one  second  to  catch  most  every  detail  during 
simulation  runs.  The  time  step  of  the  model  was  set  to  one  second,  which  meant  that  each 
time  step  corresponded  to  one  second  in  real  time. 

B.  SCENARIOS 

We  built  our  scenarios  by  taking  historical  incidents  as  a  reference.  Terrorist 
groups  have  conducted  bloody  attacks  against  Turkish  military  assets,  police  forces,  and 
civilians  along  the  Turkish-Iraqi  border  in  the  past  causing  many  deaths.  We  gave  some 
examples  of  these  attacks  in  Chapter  I.  Sometimes  terrorist  groups  attacked  with  only  a 
couple  of  terrorists  and  sometimes  made  simultaneous  coordinated  attacks  in  different 
locations  with  many  different  groups.  One  of  these  examples  occurred  on  September  11, 
2011  in  Semdinli  province  of  southeastern  Turkey.  A  total  of  130  PKK  terrorists  attacked 
Semdinli’s  District  Police  Headquarters,  District  Gendarmerie  command,  3rd  Tactical 
Mountaineer  Gendarmerie  Brigade,  and  police  checkpoint  simultaneously  with  machine 
guns  and  rocket  launchers  [48].  Three  people  were  killed,  and  seven  were  injured  in  this 
attack.  Turkish  military  and  police  forces  immediately  responded  to  the  attacks;  seven 
terrorists  were  killed,  and  the  rest  ran  away  towards  northern  Iraq. 

Our  scenario  is  based  on  this  incident.  By  experiencing  this  horrible  terrorist 
attack,  we  were  reminded  that  preventing  terrorist  groups  from  crossing  the  border  is 
crucial.  Otherwise,  a  skirmish  between  our  soldiers  and  terrorists  and  resulting  deaths  are 
inevitable.  We  are  trying  to  figure  out  how  effective  UCAVs  might  be  in  that  area  as  an 
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alternative  to  F-16s.  For  this  purpose,  we  constructed  eight  different  air-to-ground  attack 
plans  comprising  different  combinations  of  F-16s,  UAVs,  and  UCAVs.  We  call  them 
different  scenarios  throughout  the  thesis  since  each  of  them  needs  to  be  run  separately  in 
MANA,  one  baseline  and  seven  alternative  scenarios. 

1.  Baseline  Scenario 

The  baseline  scenario  was  constructed  based  on  the  skirmish  explained  above. 
This  scenario  represents  current  counterterrorism  methods  of  the  Turkish  Air  Force.  Our 
baseline  scenario  includes  three  F- 16s  as  strike  assets,  three  UAVs  for  ISR  missions,  and 
three  terrorist  groups  that  attempt  to  cross  the  border  and  make  their  way  to  attack  three 
blue  battalions.  The  baseline  scenario  also  has  partisans,  who  are  the  supporters  of 
terrorists  in  the  cities,  and  a  dummy  agent,  who  is  used  to  terminate  the  runs  after  a 
terrorist  reaches  their  objective.  As  mentioned  previously,  the  battle  area  is  separated  into 
three  regions  and  each  F-16,  UAV,  and  terrorist  group  are  assigned  to  a  specific  region. 
Figure  9  depicts  the  general  overlay  of  the  baseline  scenario  and  its  three  regions.  In  this 
scenario,  if  a  UAV  detects  and  classifies  any  terrorist  activity  along  the  border,  it 
immediately  reports  it  to  the  F-16  that  is  responsible  for  that  specific  region.  Each  UAV 
and  F-16  conduct  operations  in  a  dedicated  region,  and  they  do  not  have  any  activity  in 
other  regions. 


29 


Figure  9.  Baseline  scenario. 


2.  Alternative  Scenario  1 

In  this  alternative  scenario  (see  Figure  10),  we  replace  all  F-16s  with  UCAVs. 
UAVs  still  conduct  ISR  missions  in  their  regions.  Similar  to  baseline  scenario,  in 
alternative  scenario  1,  each  UCAV  is  responsible  for  one  specific  region.  The  biggest 
difference  between  the  baseline  scenario  and  alternative  scenario  1  is  that  in  alternative 
scenario  1  we  double  our  detection  sensors  with  the  addition  of  UCAVs.  UCAVs  also 
search  for  terrorist  activities  along  the  border  in  conjunction  with  UAVs,  and  if  they  find 
a  terrorist  group,  they  can  quickly  conduct  an  attack  mission  against  them. 
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Figure  10.  Alternative  scenario  1. 


3.  Alternative  Scenarios  2,  3,  and  4 

In  alternative  scenarios  2,  3,  and  4,  we  use  one  F-16  and  two  UCAVs  as  air-to- 
ground  strike  assets.  We  assign  those  assets  to  different  regions  in  each  scenario.  In 
alternative  scenario  2,  the  F-16  is  responsible  for  region  1  and  UCAVs  are  responsible  for 
regions  2  and  3.  In  alternative  scenarios  3  and  4,  the  F-16  is  assigned  to  regions  2  and  3, 
respectively.  We  keep  UAVs  active  in  these  scenarios,  like  we  do  in  our  other  alternative 
scenarios.  Therefore,  two  of  the  three  regions  have  two  detection  sensors  (the  sensors  on 
the  UAVs  and  UCAVs)  while  one  of  the  regions  has  only  one  sensor,  the  UAV’s  sensor. 
As  a  result,  we  expect  the  reaction  time  to  be  smaller  in  the  single-sensor  regions  than  the 
reaction  time  in  the  double-sensor  region.  Figure  1 1  shows  alternative  scenario  3  as  an 
example  of  one  of  the  F-16  and  two  UCAVs  scenarios. 
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Figure  11.  Alternative  scenario  3. 

4.  Alternative  Scenarios  5,  6,  and  7 

In  alternative  scenarios  5,  6,  and  7,  we  use  two  F-16s  and  one  UCAV  as  air-to- 
ground  strike  assets.  We  assign  those  assets  to  different  regions  in  each  scenario.  In 
alternative  scenario  5,  F-16s  are  responsible  for  regions  1  and  2,  whereas  a  UCAV  is 
responsible  for  region  3.  In  alternative  scenarios  6  and  7,  the  lone  UCAV  is  assigned  to 
regions  2  and  1,  respectively.  We  keep  UAVs  active  in  these  scenarios  just  as  we  did  for 
the  other  alternative  scenarios.  Therefore,  one  of  the  regions  among  three  have  two 
detection  sensors  (UAVs  and  UCAVs)  while  the  other  two  regions  have  only  one  sensor, 
the  UAV’s  sensor.  As  a  result,  we  expect  the  reaction  time  to  be  larger  in  single-sensor 
regions  than  the  reaction  time  in  double-sensor  region.  Figure  12  shows  alternative 
scenario  7  as  an  example  of  a  two  F-16s  and  one  UCAV  scenario. 
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Figure  12.  Alternative  scenario  7. 


C.  MANA  USER  INTERFACE 

After  defining  the  battlefield,  constructing  necessary  maps,  and  building  the  basic 
scenario,  the  user  can  edit  agent  properties  using  MANA’s  graphical  user  interface 
(GUI).  The  “edit  squad  properties”  tab  under  the  “setup”  menu  provides  access  to  squad 
properties.  The  user  can  change  various  properties  of  a  squad  by  selecting  tabs  under  the 
“edit  squad  properties”  window.  These  tabs  include  general,  map,  personalities,  tangibles, 
sensors,  weapons,  intra-squad  SA,  inter-squad  SA,  and  advanced.  The  tabs  in  the  “edit 
squad  properties”  window  are  shown  in  Figure  13.  Before  explaining  these  tabs,  we 
explain  the  trigger  states  first.  The  MANA  manual  defines  trigger  states  as  follows. 

For  a  simple  MANA  scenario,  personality  weightings  and  other  agent 
characteristics  are  first  defined  for  each  squad’s  Default  state.  This  may  be 
all  that  is  required  for  a  simple  scenario.  Once  a  scenario  starts  running 
MANA  has  the  flexibility  to  allow  agents  to  trigger  into  other  states  apart 
from  the  Default  state  depending  on  different  events  occurring  on  the 
battlefield.  This  adds  a  new  level  of  richness  to  the  types  of  agent  behavior 
which  might  emerge  in  a  scenario.  [39] 

The  “general”  tab  is  the  place  where  general  squad  data  is  defined.  The  user  can 
change  a  squad’s  name,  number  of  agents,  and  initial  fuel  capacity  by  using  this  tab.  The 
user  can  also  activate  or  deactivate  a  squad  from  here  for  a  specific  replication  of  the 
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simulation.  The  “general”  tab  also  allows  the  user  to  save,  delete,  or  load  a  squad.  The 
“map”  tab  is  the  place  where  the  user  defines  the  home  and  waypoints  of  a  squad.  The 
information  in  both  general  and  map  tabs  does  not  vary  as  a  function  of  trigger  states. 

The  “personality”  tab  allows  the  user  to  define  a  set  of  personality  weightings  for 
a  squad  of  agents  using  slider  bars,  as  illustrated  in  Figure  13.  The  slider  bars  on  the 
panel  are  for  adjusting  agent  propensities.  A  positive  value  means  an  agent  has  a  positive 
propensity  towards  the  associated  object  while  a  negative  value  indicates  a  negative 
propensity.  The  higher  the  weighting  value  is,  the  greater  the  attraction.  For  example,  in 
Figure  13  there  is  a  positive  movement  weighting  of  100  toward  enemy  agents.  There  are 
three  sources  from  which  the  information  originates.  The  information  that  agents  collect 
directly  using  their  own  sensors  corresponds  to  the  “agent  SA”  section.  The  shared  squad 
information  corresponds  to  the  “squad  SA”  section.  Inorganic  SA  refers  to  the 
information  that  has  been  obtained  from  other  squads  via  communications  links. 
Different  trigger  states  can  have  different  settings. 


Figure  13.  Personality  settings  panel. 
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The  “tangibles”  tab  allows  the  user  to  adjust  an  agent’s  speed,  ieon  type  and  eolor, 
allegianee,  threat  level,  and  class.  The  user  can  also  define  the  number  of  shots  required 
to  kill  an  agent  and  armor  thickness  using  the  “tangibles”  tab.  For  more  detailed 
information,  please  refer  to  the  MANA  manual  [39].  The  “tangibles”  tab  in  the  “edit 
squad  properties”  window  is  shown  in  Figure  14. 
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Figure  14.  Tangibles  panel. 


The  “sensors”  panel,  shown  in  Figure  15,  allows  the  user  to  define  up  to  six 
different  sensors  for  an  agent.  The  user  can  select  any  sensor,  and  change  its  name  and 
use  it  in  simple  or  advanced  mode.  The  range  tables  allow  the  user  to  adjust  the  detection 
and  classification  ranges  of  a  sensor.  The  user  also  defines  an  average  time  between 
detections  and  a  classification  probability  for  a  sensor  in  this  panel.  The  user  can  define  a 
single  range,  time,  and  probability  value  or  range  tables  to  define  the  change  in  sensing 
capability  as  a  function  of  the  distance.  The  target  classes  section  allows  the  user  to 
specify  target  classes  for  this  sensor.  Sensor  characteristics  can  vary  depending  on  the 
agent’s  trigger  state. 
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Figure  15.  Sensors  panel. 


The  “weapons”  tab  consists  of  information  about  an  agent’s  weapons.  Similar  to 
sensors,  the  user  can  define  up  to  six  different  weapons  for  each  agent.  Each  weapon’s 
characteristics  can  be  set  up  differently  using  this  panel.  The  user  can  define  a  weapon’s 
class,  model,  and  fire  mode  as  well  as  the  number  of  ammunition  an  agent  has.  Similar  to 
the  “sensors”  panel,  there  is  a  range-probability  table  for  each  weapon  to  define  the 
probability  of  hitting  a  target  at  a  corresponding  distance.  In  another  table,  the  user 
specifies  target  and  non-target  classes.  Weapon  characteristics  can  vary  depending  on  the 
agent’s  trigger  state.  The  user  can  also  enable  or  disable  a  weapon  from  this  panel.  The 
“weapons”  tab  in  the  “edit  squad  properties”  window  is  shown  in  Figure  16.  For  other 
sections  that  we  did  not  explain  here,  please  refer  to  the  MANA  manual  [39]. 
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Figure  16.  Weapons  panel. 


The  “intra-squad”  and  the  “inter-squad  SA”  tabs  refer  to  how  the  information  is 
shared  within  the  squad  and  between  the  squads,  respectively.  By  using  the  “intra-squad 
SA”  tab,  the  user  defines  communication  delays  between  the  agents  within  a  squad  as 
well  as  the  squad  contact  persistence.  The  “inter-squad”  tab,  depicted  in  Figure  17,  allows 
the  user  to  define  a  communication  link  between  any  two  squads.  The  user  can  change 
the  maximum  communication  range,  latency,  maximum  capacity,  and  reliability  of 
communication  for  each  agent  or  squad  using  this  panel.  The  allegiance,  location, 
velocity,  and  movement  direction  of  an  agent  can  be  transferred  through  these  links. 
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Figure  17.  Inter-squad  SA  panel. 


Finally,  the  “advaneed”  tab  provides  the  user  more  sophistieated  information  such 
as  the  random  patrol  tendency  of  an  agent,  formation  type  of  a  squad,  and  the  search 
algorithm  type  of  an  agent. 

D.  AGENT  DESCRIPTIONS 

The  primary  element  of  the  MANA  model  is  the  squad.  A  squad  can  be  either  a 
single  agent  or  a  group  of  agents.  When  we  define  each  squad’s  allegiance  in  MANA,  it 
affects  a  squad’s  behavior  when  it  interacts  with  other  squads.  Table  1  displays  the  17 
squads  used  in  this  model. 

Allegiance  determines  which  side  of  the  battle  an  agent  will  fight  on.  There  are 
three  options  for  an  agent’s  allegiance:  enemy,  friend,  or  neutral.  Neutral  can  be  used  to 
represent  civilians  or  noncombatants  in  a  scenario.  Agent  threat  levels  and  agent  class  are 
used  to  delineate  between  enemy  types  in  a  scenario.  For  example,  the  user  can 
differentiate  between  heavy-armored  vehicles  or  light  infantry  by  defining  heavy- 
armored  vehicles  as  threat-level  three  and  light  infantry  as  threat-level  two. 
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Table  1.  Allegiance,  threat  level,  and  class  of  squads. 


Squad  No 

Squad  Name 

Allegiance 

Threat  Level 

Class 

1 

Blue  UAVl 

1 

3 

1 

2 

Blue  UAV2 

1 

3 

1 

3 

Blue  UAVS 

1 

3 

1 

4 

F-16  1 

1 

3 

1 

5 

F-16 2 

1 

3 

1 

6 

F-16  3 

1 

3 

1 

7 

UCAVl 

1 

3 

1 

8 

UCAV2 

1 

3 

1 

9 

UCAVS 

1 

3 

1 

10 

Blue  Bat  1 

1 

3 

1 

11 

Blue Bat 2 

1 

3 

1 

12 

BlueBatS 

1 

3 

1 

13 

RedTeaml 

2 

3 

2 

14 

Red  Team2 

2 

3 

2 

15 

RedTeamS 

2 

3 

2 

16 

Partisan 

2 

0 

0 

17 

Dummy  Agent 

2 

3 

99 

Figure  18  depicts  the  starting  positions  of  the  agents  on  the  battlefield.  The  yellow 
line  represents  the  Turkish-Iraqi  border.  Three  terrorist  groups  are  located  south  of  the 
border  while  UAVs  and  UCAVs  are  located  to  the  north.  F-16s  are  waiting  to  be  called  at 
the  top  left  comer,  where  the  symbolic  air  force  base  is  located.  Terrorist  partisans  are 
located  around  the  air  force  base,  so  they  can  provide  the  terrorists  with  information  on 
aircraft  activity. 


Figure  18.  Overview  of  the  battlefield  with  all  agents  included. 
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1. 


Blue  UAVs 


Blue  UAVs  are  assigned  to  intelligence  surveillance  and  reconnaissance  missions 
in  a  designated  region  along  the  Turkish-Iraqi  border.  Simulated  UAVs  in  this  thesis  are 
medium-altitude,  long-endurance  UAVs  capable  of  flying  over  24  hours.  Blue  UAVs 
follow  predetermined  waypoints  after  the  simulation  starts.  They  are  basically 
programmed  to  fly  back  and  forth  along  the  border  section  to  which  they  are  assigned  in  a 
“default”  state.  Using  their  onboard  sensors,  the  UAVs  try  to  detect  and  classify  terrorists 
if  they  exist.  Once  they  detect  any  activity  along  the  border,  they  focus  on  this  location 
and  try  to  classify  the  detected  activity.  If  the  UAVs  confirm  a  terrorist  presence,  they  go 
into  an  “enemy  contact”  state  and  report  it  to  the  weaponized  units  through 
communication  links  after  a  user  input  delay.  This  time-delay  accounts  for  the  time  spent 
during  conversation  and  the  reaction  time.  This  process  is  summarized  in  Table  2. 


Table  2.  State  process  of  blue  UAV  squads  in  MANA. 


State 

Positive 

propensity 

Negative 

propensity 

Speed 

Duration 

(sec) 

Fallback 

state 

Communication 

link 

Run  start 

Next  waypoint 
Unknowns 

- 

0 

1 

Default 

F-16  squads 
UCAV  squads 

Default 

Next  waypoint 
Unknowns 

- 

90  knots 

- 

- 

Enemy  contact  3 

Enemy  threat  3 
Unknowns 

- 

90  knots 

5000 

Default 

2.  F-16s 

There  are  three  F-16s  located  at  the  Air  Force  base.  They  are  configured  with  air- 
to-ground  weapons  as  well  as  with  sensors.  F-16s  do  not  act  until  they  receive  a  scramble 
order  as  a  result  of  a  terrorist  detection  by  a  UAV.  A  scramble  response  time  of  5  to  15 
minutes  for  F- 16  squads  accounts  for  the  communication  latency  between  them  and  the 
UAVs.  After  UAVs  detect  terrorist  activity,  F-16s  go  into  an  “inorganic  SA  enemy 
contact”  state  and  fly  directly  to  the  region.  Afterward,  using  their  onboard  sensors,  they 
detect  and  classify  the  terrorist  group,  based  on  the  UAV  report,  which  means  they  are  in 
an  “enemy  contact”  state  and  conduct  an  air-to-ground  attacks. 
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Each  F-16  can  carry  two  weapons.  After  launching  the  first  bomb,  they  go  into  an 
“ammo  out  weapon  1”  state  in  which  a  five-minute  delay  accounts  for  a  repositioning 
time  for  the  F-16s.  F-16s  have  two  sensors,  one  for  terrorists  and  the  other  for  dummy 
agents.  They  also  have  three  weapons,  two  of  which  will  be  used  against  terrorists  and 
one  against  dummy  agents. 

Effective  ranges  of  F-16  sensors  and  weapons  in  the  simulation  are  greater  than 
their  real-world  values.  The  reason  for  this  is  the  following.  Real-world  terrorist 
formations  are  normally  very  small  compared  to  the  battlefield,  and  during  an  attack,  they 
typically  scatter  in  a  2  to  3  km  area.  But  if  we  modeled  our  scenario  this  way,  we  would 
see  an  entire  terrorist  group  as  a  dot  on  the  battlefield.  Therefore,  we  dispersed  them  10  to 
15  km  to  see  actions  and  make  modifications  to  the  model  more  easily.  Finally,  we 
increased  detection,  classification,  and  weapon  ranges  proportionally  as  well  in  order  to 
get  a  realistic  result. 

After  dropping  two  bombs,  F- 16s  go  into  the  “ammo  out  weapon  2”  state,  return 
to  a  designated  area  where  a  dummy  agent  is  located,  and  kill  the  dummy  agent  to  halt 
the  scenario  run.  The  scenario  terminates  when  the  dummy  agent  takes  three  hits.  If  F- 16s 
run  out  of  fuel  before  they  drop  their  second  weapon,  they  suspend  their  missions  for  25 
minutes,  which  corresponds  with  the  time  spent  during  air-to-air  refueling.  This  process 
is  summarized  in  Table  3. 


Table  3.  State  process  of  F-16  squads  in  MANA. 


State 

Positive 

propensity 

Negative 

propensity 

Speed 

Duration 

(sec) 

Fallback 

state 

Communication 

link 

Run  start 

Next  waypoint 

- 

0  knots 

99999 

Default 

UAV  squads 

Default 

Enemy  threat  3 

- 

300  knots 

- 

- 

Enemy  contact  3 

Enemy  threat  3 

- 

300  knots 

10000 

Default 

Inorganic  SA 
Enemy  contact  3 

Enemy  threat  3 

- 

150  knots 

99999 

Spare  1 

Spare  1 

Enemy  threat  3 

- 

300  knots 

99999 

Default 

Ammo  out 
weapon  1 

Enemy  threat  3 

- 

300  knots 

300 

Spare  1 

Fuel  out 

Enemy  threat  3 

- 

0  knots 

1500 

Spare  2 

Spare  2 

Enemy  threat  3 

- 

0  knots 

1 

Default 

Ammo  out 
weapon  2 

Next  waypoint 

- 

2000 

knots 

99999 

Default 
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3. 


UCAVs 


There  are  three  UCAVs  assigned  to  a  specific  portion  of  the  border.  Similar  to 
UAVs,  UCAVs  start  searching  for  terrorists  along  the  border  in  their  “default”  state. 
However,  different  than  the  UAVs,  they  can  carry  two  air-to-ground  weapons,  like  the  F- 
16s.  When  they  detect  and  classify  terrorists,  they  go  into  an  “enemy  contact”  state. 
Thereafter,  they  drop  their  first  weapons  and  go  into  the  “ammo  out  weapon  1”  state  in 
which  a  five-minute  delay  accounts  for  a  repositioning  time  for  the  UCAVs.  After 
dropping  their  second  bombs,  they  go  into  the  “ammo  out  weapon  2”  state  and  fly  to  a 
designated  area  where  a  dummy  agent  is  located.  UCAVs  can  also  communicate  with 
UAVs.  If  UAVs  detect  terrorist  activity  first,  they  report  it  to  the  UCAVs  who  then  come 
and  launch  their  air-to-ground  weapons. 

Sensor  and  weapon  settings  of  UCAV  squads  are  similar  to  those  of  F- 16s,  but  the 
ranges  are  shorter.  UCAVs  have  two  sensors,  one  for  terrorists  and  the  other  for  dummy 
agents.  They  also  have  three  weapons,  two  of  which  are  used  against  terrorists  and  one 
against  dummy  agents.  Similar  to  F-16  squads,  we  re-scaled  the  sensor  and  weapon 
ranges  of  UCAV  squads  in  order  to  get  a  realistic  result.  This  process  is  summarized  in 
Table  4. 


Table  4.  State  process  of  UCAV  squads  in  MANA. 


State 

Positive 

propensity 

Negative 

propensity 

Speed 

Duration 

(sec) 

Fallback 

state 

Communication 

link 

Run  start 

Next  waypoint 
Unknowns 

- 

100  knots 

1 

Default 

UAV  squads 

Default 

Next  waypoint 
Unknowns 

- 

100  knots 

- 

- 

Enemy  contact  3 

Enemy  threat  3 
Unknowns 

- 

90  knots 

5000 

Default 

Inorganic  SA 
Enemy  contact  3 

Enemy  threat  3 

- 

100  knots 

99999 

Spare  1 

Spare  1 

Enemy  threat  3 

- 

90  knots 

99999 

Default 

Ammo  out 
weapon  1 

Enemy  threat  3 

- 

90  knots 

300 

Spare  1 

Fuel  out 

Enemy  threat  3 

- 

0  knots 

1500 

Spare  2 

Ammo  out 
weapon  2 

Alt  waypoint 

- 

2000 

knots 

99999 

Default 
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4. 


Blue  Battalions 


Blue  battalions  in  this  model  are  symbolic  agents,  so  they  do  not  have  any  active 
mission.  There  are  three  blue  battalion  agents  that  provide  visual  information  about  the 
final  waypoints  of  terrorists.  Terrorists  try  to  reach  blue  battalions  and  attack  simulated 
military  facilities. 

5.  Red  Teams 

There  are  three  terrorist  groups  in  the  model.  Each  terrorist  group  consists  of  40 
terrorists  whose  main  objective  is  to  cross  the  border,  reach  military  facilities,  and  attack 
them.  These  groups  are  initially  located  south  of  the  Turkish-Iraqi  border  and  start  to 
proceed  to  the  north  when  the  runs  begin.  We  adjust  personality  settings  of  terrorist 
groups  so  that  they  stick  together  while  they  are  proceeding.  As  we  mentioned  before, 
each  trigger  state  can  have  different  settings.  In  the  “default”  state,  terrorists  move 
towards  their  next  waypoint  with  a  70%  propensity  while  they  conceal  at  a  value  of  40%. 
When  any  of  the  terrorists  inside  the  group  detects  a  blue  agent,  it  goes  into  an  “enemy 
contact”  state  and  informs  other  agents  in  the  group.  As  a  result,  all  agents  in  this  specific 
group  go  into  a  “squad  enemy  contact”  state  and  implement  this  state’s  personality 
settings.  In  an  “enemy  contact”  or  “squad  enemy  contact”  state,  terrorists  increase  their 
concealments  to  prevent  blue  agents  from  detecting  them.  In  order  to  increase  their 
concealment,  terrorists  tend  to  use  the  terrain  types  that  provide  more  concealment,  such 
as  moving  toward  “green”  regions  on  the  terrain  map,  which  represents  deep  valleys. 
Therefore,  they  start  to  separate  from  each  other  a  little  bit. 

Another  trigger  state  that  terrorists  can  get  into  is  an  “inorganic  SA  enemy 
contact”  state.  Terrorists  go  into  this  state  after  partisan  agents  detect  aircraft  activity  and 
deliver  this  information  to  terrorists  with  some  delay.  Terrorists  increase  their 
concealment  in  an  “inorganic  SA  enemy  contact”  state  as  well.  A  “squad  shot  at  (sec)” 
state  means  that  F-16s  or  UCAVs  have  dropped  their  second  weapons  on  the  terrorists.  In 
this  state  terrorists  turn  back,  and  run  away  toward  the  mountains  and  hide  inside  the 
caves. 
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Terrorists  can  detect  and  classify  enemy  agents  within  a  defined  range.  The 
elevation  map  affects  detection  and  classification  capabilities  of  terrorists.  Terrorists  have 
weapons  that  can  fire  within  a  certain  range.  They  will  use  this  weapon  against  imaginary 
blue  soldiers  inside  the  blue  battalions.  Because  there  are  no  blue  soldiers  in  the  model, 
the  number  of  terrorists  who  have  reached  the  blue  battalions  before  the  simulation  ends 
is  determined  by  the  number  of  blue  soldiers  killed.  This  process  is  summarized  in 
Table  5. 


Table  5.  State  process  of  red  team  squads  in  MANA. 


State 

Positive 

propensity 

Negative 

propensity 

Speed 

Duration 

(sec) 

Fallback 

state 

Communication 

link 

Run  start 

Next  waypoint 

- 

50  km/h 

100 

Default 

Partisan  squad 

Default 

Next  waypoint 
Concealment 

Uninjured 

Friends 

Injured  Friends 

3  km/h 

- 

- 

Enemy  contact  3 

Next  waypoint 
Concealment 
Cover 

- 

2  km/h 

1500 

Default 

Inorganic  SA 
Enemy  contact  3 

Next  waypoint 
Concealment 
Cover 

- 

2  km/h 

3000 

Default 

Squad 

Enemy  contact  3 

Next  waypoint 
Concealment 
Cover 

- 

2  km/h 

1500 

Default 

Squad  shot  at 
(sec) 

Alt  waypoint 
Concealment 

- 

5  km/h 

99999 

Default 

6.  Partisans 

Partisans  are  the  civil  supporters  of  terrorist  groups  who  live  in  the  cities  where 
the  AFB  is  located.  When  they  see  aircraft  activity  inside  the  AFB,  they  inform  terrorists 
about  this,  so  terrorists  can  hide  or  take  some  precautions.  Partisans  communicate  with 
terrorist  groups  through  communication  links  which  are  set  using  the  “inter-squad  SA” 
tab  in  MANA. 


7.  Dummy  Agent 

Dummy  agents  are  only  used  to  terminate  each  run.  Each  strike  asset,  either  an  F- 
16  or  a  UCAV  depending  on  the  scenario,  carries  a  weapon  that  can  be  employed  against 


44 


dummy  agents.  After  finishing  their  missions,  F-16s  and  UCAVs  fly  to  the  dummy 
agent’s  eoordinate  and  drop  their  speeial  weapons.  A  dummy  agent  is  killed  after  it  takes 
three  shots,  thereby  terminating  the  seenario  (i.e.,  all  three  terrorist  groups  have  been 
attacked  and  stopped). 


45 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


46 


IV.  EXPERIMENTAL  DESIGN 


Design-of-experiment  (DOE)  techniques  allow  the  user  to  vary  the  input 
parameters  in  an  efficient  manner  and  to  generate  multiple  variations  of  a  model  [49]. 
The  user  can  explore  the  effects  of  the  factors  used  in  the  model  as  well  as  the 
interactions  between  the  factors  by  using  DOE  techniques.  DOE  is  a  critical  part  of  the 
modeling  and  simulation  analysis  process.  In  this  chapter,  we  explain  the  factors  used  to 
construct  the  experimental  design  of  our  study.  Next,  we  explain  the  nearly  orthogonal 
Latin  hypercube  (NOLH)  design  that  is  used  in  this  thesis.  Last,  we  mention  the  model 
running  process. 

A.  FACTORS  OF  INTEREST 

There  are  numerous  factors  that  affect  the  outcome  of  a  military  operation  in  the 
real  world.  We  can  divide  these  factors  into  two  groups:  controllable  and  uncontrollable. 
Controllable  factors,  also  known  as  decision  variables,  are  those  that  the  user  can 
manipulate  in  the  real  world  during  the  system  development  or  the  employment  phase. 
For  instance,  aircraft  speed,  sensor  detection  range,  or  aircraft  altitude  are  controllable 
factors.  Uncontrollable  factors  are  those  that  the  user  cannot  manipulate  in  the  real  world. 
Enemy  characteristics,  such  as  number  of  terrorists  and  enemy  weapon  capabilities,  or 
environmental  characteristics,  such  as  wind  speed,  can  be  considered  uncontrollable 
factors.  It  is  impossible  to  take  into  account  every  factor  in  a  simulation  environment. 
Therefore,  we  varied  the  28  factors  that  we  believe  will  influence  the  outcome  the  most. 
Among  these  28  factors,  20  are  controllable  factors  and  eight  are  uncontrollable  factors. 

1.  Controllable  Factors 

In  this  thesis,  controllable  factors  include  the  parameters  of  F- 16s,  UCAVs,  and 

UAVs.  We  varied  the  sensor,  weapon,  and  time-related  parameters  of  F-16s,  UCAVs, 

and  UAVs  within  some  ranges  to  explore  the  effects  on  the  outcome.  The  factor  names 

are  the  same  as  those  produced  in  the  MANA  output.  Table  6  provides  a  general 

explanation  of  controllable  factors  and  their  ranges.  The  following  sections  explain  the 

factors  and  the  reasons  of  selecting  minimum  and  maximum  values  associated  to  those 
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factors  in  detail.  Because  we  use  the  same  factors  for  F-16s,  UCAVs,  and  UAVs,  we 
explain  the  factor  and  the  differences  among  the  platforms  for  that  factor  in  the  same 
section. 


Table  6.  Controllable  factors,  brief  explanations,  minimum  and  maximum 
ranges,  and  units  of  the  factors  used  in  the  model. 


Agents 

Factors 

Explanation 

Min 

Max 

Unit 

F16SnrClsRng 

Sensor  classification  range  in  the  default  state 

4000 

8000 

meter 

F 1 6SnrDetToClassRatio 

Sensor  detection  to  classification  range  ratio  in  the 
default  state 

1 

2 

VC 

F16ClsProb 

Classification  probability  in  the  default  state 

0.6 

0.9 

F 1 6T  imeBetv^  Det 

Average  time  bet\\  een  detections  in  the  default  state 

100 

800 

second 

F16WpnRng 

Maximum  effective  weapon  range 

4000 

8000 

meter 

F16HitProb 

Weapon  hit  probability  at  maximum  weapon  range 

0.4 

0.8 

F16ScrTime 

Scramble  response  time 

420 

900 

second 

UCAVSnrClsRng 

Sensor  classification  range  in  the  default  state 

6000 

8000 

meter 

UCAV  SnrDetT  oClassRatio 

Sensor  detection  to  classification  range  ratio  in  the 
default  state 

1 

2 

UCAVClsProb 

Classification  probability  in  the  default  state 

0.1 

0.7 

UCAVT  imeBetv\  Det 

Average  time  betw  een  detections  in  the  default  state 

400 

1000 

second 

w 

P 

UCAWpnRng 

Maximum  effective  weapon  range 

3000 

5000 

meter 

UCA\HitProb 

Weapon  hit  rate  at  maximum  weapon  range 

0.5 

0.8 

UCAVScrTime 

Communication  delay  with  UAV 

100 

600 

second 

UCAVSpd 

Velocity  in  the  default  state 

90 

200 

kts 

UAVSnrClsRng 

Sensor  classification  range  in  the  default  state 

4000 

8000 

meter 

> 

UAVSnrDetToClassRatio 

Sensor  detection  to  classification  range  ratio  in  the 
default  state 

1 

2 

< 

UAVClsProb 

Classification  probability  in  the  default  state 

0.07 

0.5 

UAVT  imeBetv^Uet 

Average  time  betw  een  detections  in  the  default  state 

700 

1300 

second 

UAVSpd 

Velocity  in  the  default  state 

80 

110 

kts 

a.  Sensor  Classification  Range 

The  sensor  classification  range  (SnrClsRng)  applies  for  F-16,  UCAV,  and  UAV 
squads.  Sensor  classification  range  is  the  distance  at  which  an  F-16,  UCAV,  and  UAV 
can  classify  other  squads. 

For  F-16s,  this  value  varies  from  4000  to  8000  meters.  Classification  range  of  an 
F-16  is  greater  than  its  real-world  value  because,  in  the  real  world,  terrorists  are  normally 
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very  small  compared  to  the  battlefield.  During  an  attack,  they  normally  scatter  in  a  2  to  3 
km  area,  but  if  we  modeled  our  scenario  this  way,  we  would  see  one  terrorist  group  as  a 
dot  on  the  battlefield.  Therefore,  we  dispersed  them  10  to  15  km  to  see  actions  and  make 
modifications  on  the  model.  Finally,  we  increased  the  classification  range  proportionally 
in  order  to  get  a  realistic  result.  The  same  rule  applies  for  UCAV  and  UAV  squads  as 
well. 

For  UCAVs,  this  value  varies  from  6000  to  8000  meters,  which  is  greater  than 
those  for  F-16s.  This  difference  is  because  of  two  reasons.  First,  the  UCAV  that  we  use  in 
our  model  will  have  more  sophisticated  sensors.  Second,  based  on  our  experience  and 
previous  operations,  it  is  more  difficult  for  a  pilot  to  classify  terrorists  while  flying  the  jet 
than  for  a  remote  operator  of  a  UCAV.  Therefore,  we  give  the  advantage  to  the  UCAV 
for  classifying  the  terrorists. 

For  UAVs,  this  value  varies  from  4000  to  8000  meters,  which  is  the  same  of  those 
for  F-16s.  This  is  because  UAVs  that  we  simulate  in  this  thesis  are  old  compared  to  the 
UCAVs. 


b.  Sensor  Detection  to  Classification  Range  Ratio 

The  sensor  detection  to  classification  range  ratio  (SnrDetToClassRatio)  applies 
for  F-16,  UCAV,  and  UAV  squads.  This  ratio  is  the  value  obtained  by  dividing  the  sensor 
detection  range  by  a  factor  to  get  the  sensor  classification  range. 

The  sensor  detection  range  must  be  less  than  or  equal  to  the  sensor  classification 
range  because  a  sensor  cannot  classify  a  threat  without  first  detecting  it.  Therefore,  we 
vary  the  sensor  detection  to  classification  ratio  from  1  to  2  for  F-16s,  UCAVs,  and  UAVs. 
We  multiply  this  number  with  the  classification  range  to  determine  the  detection  range 
that  MANA  will  use  for  calculations.  By  utilizing  this  feature,  we  prevent  MANA  from 
producing  a  detection  range  that  is  smaller  than  a  classification  range. 

c.  Classification  Probability 

The  sensor  classification  probability  (ClsProb)  applies  for  F-16,  UCAV,  and  UAV 
squads.  The  sensor  classification  probability  refers  to  how  likely  is  it  for  an  F-16,  UCAV, 
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and  UAV  to  classify  a  terrorist.  For  this  research,  we  ignored  false-positive  classification 
probabilities,  which  is  when  an  agent  wrongly  classifies  another  agent  as  a  terrorist.  The 
classification  probability  depends  on  several  factors  in  the  real  world,  such  as  sensor 
capabilities,  pilot  capabilities,  weather,  and  terrain. 

Based  on  personal  experiences  and  previous  operations,  we  vary  the  classification 
probability  of  an  F-16  from  0.6  to  0.9.  This  range  seems  higher  than  the  real  world 
values.  The  reason  for  that  is  the  following.  F-16  squads  first  interact  with  terrorist 
groups  after  UAVs  detect  and  classify  them.  Therefore,  F-16s  deploy  to  the  area  with  the 
location  information  of  the  terrorist  groups  and  start  searching  for  terrorists  based  on  this 
information.  As  a  result,  it  is  more  likely  for  an  F-16  to  classify  a  terrorist  group  than 
other  units  in  the  model. 

We  vary  the  classification  probability  from  0.1  to  0.7  and  from  0.07  to  0.5  for 
UCAVs  and  UAVs,  respectively.  The  difference  between  the  UCAV  and  the  UAV 
classification  probability  is  because  UCAVs  have  more  sophisticated  sensors  than  UAVs 
in  our  model. 


d.  Average  Time  Between  Detection 

The  average  time  between  detection  (TimeBetwDet)  applies  for  F-16,  UCAV,  and 
UAV  squads.  The  average  time  between  detections  factor  corresponds  to  the  detection 
capability  of  an  F-16,  UCAV,  and  UAV.  Different  than  the  classification  probability, 
MANA  does  not  provide  an  option  to  explicitly  define  detection  probability  to  the  user. 
In  MANA,  the  per  time-step  detection  probability  is  set  given  the  average  time  between 
detection — as  one  gets  with  a  geometric  random  variable. 

For  F-16s,  we  vary  the  average  time  between  detections  from  100  to  800  seconds, 
whereas  we  use  400  to  1000  and  700  to  1300  seconds  for  UCAVs  and  UAVs, 
respectively.  The  average  time  between  detections  is  smaller  for  F-16s  than  those  for 
UCAVs  and  UAVs  because  F-16s  start  to  search  for  the  terrorists  with  a  location 
knowledge  provided  by  UAVs.  However,  UCAVs  and  UAVs  do  not  have  any 
preliminary  knowledge  about  the  location  of  the  terrorists.  The  reason  that  a  UACV’s 
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average  time  between  detections  is  smaller  than  those  of  a  UAV  is  UCAVs  have  more 
advanced  onboard  sensors  in  the  model. 

e.  Maximum  Effective  Weapon  Range 

The  maximum  effective  weapon  range  (WpnRng)  applies  for  F-16  and  UCAV 
squads.  The  weapon  range  is  the  maximum  effective  range  at  which  an  air-to-ground 
weapon  of  an  F-16  and  UCAV  can  kill  a  terrorist.  MANA  has  two  modes  for  weapon 
settings,  simple  and  advanced.  We  used  the  advanced  mode  in  which  weapon 
effectiveness  is  determined  by  a  probability-range  table.  We  defined  two  ranges  and 
probabilities  corresponding  to  these  ranges  in  a  table,  one  for  minimum  and  the  other  for 
maximum  range.  In  our  model,  we  only  vary  the  maximum  weapon  range  for  the 
weapons  that  an  F-16  and  a  UCAV  can  carry.  MANA  interpolates  the  ranges  and  the 
probabilities  between  the  minimum  and  maximum  values. 

Similar  to  the  sensor  ranges,  weapon  range  is  also  greater  than  real  world  values 
because  we  increased  it  proportionally  to  the  terrorist  dispersion  on  the  battlefield.  We 
vary  maximum  effective  weapon  range  of  an  F-16  from  4000  to  8000  meters.  We  reduce 
this  range  to  3000  to  5000  meters  for  UCAVs  because  the  UCAVs  that  we  simulate  in  the 
model  carry  smaller  weapons  than  those  of  the  F-16s.  Based  on  personal  experience  and 
previous  operations,  we  decided  that  these  ranges  are  appropriate  for  our  model’s  scale. 

f  Weapon  Hit  Probability 

The  weapon  hit  probability  (HitProb)  applies  for  F-16  and  UCAV  squads.  The 
weapon  hit  probability  is  the  probability  that  an  air-to-ground  weapon  of  an  F-16  and 
UCAV  hits  the  target.  Similar  to  the  weapon  range  factor,  we  only  vary  the  weapon  hit 
probability  at  the  maximum  weapon  range  and  MANA  interpolates  the  values  between 
minimum  and  maximum.  This  probability  depends  on  several  factors,  such  as  sensor 
capabilities,  weather,  target  mobility,  pilot  capability,  and  terrain. 

We  vary  this  value  from  0.4  to  0.8  and  from  0.5  to  0.8  for  F-16s  and  UCAVs, 
respectively.  We  give  UCAVs  a  small  advantage  because  they  have  more  advanced 
onboard  sensors  and  more  precise  weapons  than  F-16s  in  the  model. 
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g.  Scramble  Response  Time 

The  scramble  response  time  (ScrTime)  applies  for  F-16  and  UCAV  squads.  The 
scramble  response  has  different  meanings  in  the  model.  For  an  F-16,  it  is  the  time  in 
which  an  F-16  takes  off  after  receiving  a  scramble  order.  In  our  model,  we  utilize  this 
time  by  defining  communication  latency  between  the  F-16  and  the  UAV.  After  a  UAV 
classifies  a  terrorist  group,  it  reports  this  group  to  the  F-16  with  some  delay,  which 
corresponds  to  the  scramble  response  time  of  F-16.  We  vary  this  value  from  420  to  900 
seconds  for  an  F-16. 

For  a  UCAV,  the  scramble  response  time  is  the  time  in  which  a  UAV  can  pass  the 
information  of  terrorist  locations  to  a  UCAV.  Different  than  the  F-16s,  UCAVs  start 
searching  for  terrorists  along  the  border  like  UAVs.  If  a  UAV  detects  a  terrorist  activity 
first,  it  reports  it  to  the  UCAV.  For  this  case,  we  use  the  scramble  response  time  factor  to 
represent  a  communication  delay  between  the  UAV  and  the  UCAV  through  the 
communication  latency  in  MANA.  We  vary  this  value  from  100  to  600  seconds  for  a 
UCAV.  It  is  smaller  than  those  for  the  F-16  because  this  factor  represents  both  the 
communication  delay  and  the  reaction  time  for  an  F-16  while  it  refers  only  to  the 
communication  delay  for  a  UCAV. 

h.  Speed 

The  speed  factor  applies  for  UAV  and  UCAV  squads.  The  speed  is  the  velocity  of 
a  UAV  and  a  UCAV  during  the  search.  We  vary  this  value  from  90  to  200  knots  for  a 
UCAV,  while  we  vary  it  from  80  to  110  knots  for  a  UAV.  Again,  the  increased  speed  is 
because  UCAVs  are  newer  systems  than  UAVs. 

2.  Uncontrollable  Factors 

Uncontrollable  factors  include  the  parameters  of  terrorists  and  partisans.  We 

varied  sensor  and  time-related  parameters  of  terrorists  and  partisans  within  some  ranges 

to  explore  the  effects  on  the  outcome.  The  factor  names  are  the  same  as  those  produced  in 
MANA  output.  Table  7  provides  a  general  explanation  of  the  uncontrollable  factors  and 
their  ranges.  The  following  sections  explain  the  factors  and  the  reasons  of  selecting  the 
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minimum  and  maximum  values  associated  to  those  factors  in  detail.  Similar  to  previous 
section,  we  explain  the  factor  and  the  differences  among  the  platforms  for  that  factor  in 
the  same  section. 


Table  7.  Uncontrollable  factors,  brief  explanations,  minimum  and 
maximum  ranges,  and  units  of  the  factors  used  in  the  model. 


Ageots 

Factors 

Explanation 

Min 

Max 

Unit 

Terrorist 

TenoristClsRiig 

Sensor  classification  range  in  the  default  state 

700 

1300 

meter 

T  eiToristDetT  oClassRatio 

Sensor  detection  to  classification  range  ratio  in  the 
default  state 

1 

2 

TeiToristClsPi'ob 

Classification  probability  m  the  default  state 

0.3 

0.7 

T  eiToristTiineBetwDet 

Average  time  between  detections  in  the  default  state 

300 

700 

second 

Partisan 

PailClsRng 

Sensor  classification  range 

1500 

2500 

meter 

PailDetToClassRatio 

Sensor  detection  to  classification  range  ratio  in  the 
default  state 

1 

2 

PaiINmnber 

Niunber  of  agents 

5 

10 

PailCoiiiDelay 

Commmiication  delay  with  teiTorists 

500 

900 

second 

a.  Sensor  Classification  Range 

The  sensor  classification  range  (SnrClsRng)  applies  for  terrorist  and  partisan 
squads.  Sensor  classification  range  is  the  distance  at  which  a  terrorist  and  a  partisan  can 
classify  other  squads. 

This  value  varies  from  700  to  1300  meters  for  a  terrorist  while  it  varies  from  1500 
to  2500  meters  for  a  partisan.  We  define  this  value  larger  for  partisans  because  partisans 
live  in  the  cities  and  terrorists  are  traveling  on  a  rough  terrain,  which  decreases  the  line- 
of-sight  of  a  terrorist. 

b.  Sensor  Detection  to  Classification  Range  Ratio 

The  sensor  detection  to  classification  range  ratio  (SnrDetToClassRatio)  applies 
for  terrorist  and  partisan  squads.  This  ratio  is  the  value  obtained  by  dividing  the  sensor 
detection  range  by  a  factor  to  get  the  sensor  classification  range. 

The  sensor  detection  range  must  be  less  than  or  equal  to  the  sensor  classification 
range  because  a  sensor  cannot  classify  a  threat  without  detecting  it.  Therefore,  we  vary 
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the  sensor  detection  to  classification  ratio  from  1  to  2  for  terrorists  and  partisans.  We 
multiply  this  number  with  the  classification  range  to  determine  the  detection  range  that 
MANA  uses  for  calculations.  By  utilizing  this  feature,  we  prevent  MANA  from 
producing  a  detection  range  that  is  smaller  than  a  classification  range. 

c.  Sensor  Classification  Probability 

The  sensor  classification  probability  (TerroristClsProb)  applies  for  terrorist 
squads.  The  sensor  classification  probability  refers  to  how  likely  it  is  for  a  terrorist  to 
classify  a  blue  agent.  We  ignored  false-positive  classification  probability,  which  means  a 
terrorist  agent  classifies  a  blue  agent  as  terrorist  (or  neutral)  but  it  is  actually  not  a 
terrorist.  We  vary  the  classification  probability  of  a  terrorist  from  0.3  to  0.7. 

d.  Average  Time  Between  Detection 

The  average  time  between  detection  (TerroristTimeBetwDet)  applies  for  terrorist 
squads.  The  average  time  between  detections  factor  corresponds  to  the  detection 
capability  of  a  terrorist.  As  we  explained  under  the  average  time  between  detection 
section  of  controllable  factors,  we  use  this  factor  to  obtain  more  realistic  results  in  terms 
of  detection  time.  We  vary  this  value  from  300  to  700  seconds  for  a  terrorist. 

e.  Number  of  Partisan  Agents 

The  number  of  partisan  agents  (PartNumber)  is  the  count  of  partisans  living  in  the 
cities.  We  vary  this  value  from  5  to  10. 

f.  Partisan  Communication  Delay 

The  partisan  communication  delay  (PartComDelay)  is  the  time  in  which  a  partisan 
can  pass  information  on  aircraft  activity  to  a  terrorist  squad.  We  use  the  partisan 
communication  delay  factor  to  represent  the  communication  delay  between  the  partisans 
and  the  terrorists  through  the  communication  latency  capability  in  MANA.  We  vary  this 
value  from  500  to  900  seconds. 
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B.  NEARLY  ORTHOGONAL  LATIN  HYPERCUBE  DESIGNS 

We  explore  the  effects  of  28  different  factors  in  this  thesis.  These  are  sensor, 
weapon,  and  time-related  factors  of  the  agents.  Some  of  these  factors  have  a  wide  range 
between  their  maximum  and  minimum  levels.  It  is  impossible  to  run  our  model  for  each 
possible  combination  of  the  factor  levels  because  we  would  have  to  run  many  billions  of 
combinations,  depending  on  the  number  of  decimals  of  each  factor.  It  would  take  too 
long  to  complete  our  simulation,  even  using  a  supercomputer.  As  a  result,  we  need  to  use 
an  advanced  design  of  experiment  technique  to  reduce  the  computation  time,  but  at  the 
same  time  cover  a  broad  section  of  the  input  space.  For  this  purpose,  we  used  a  nearly 
orthogonal  Latin  hypercube  (NOTH)  design.  The  NOTH  design  was  developed  by 
Lieutenant  Colonel  Thomas  M.  Cioppa  during  his  doctoral  studies  at  the  Naval 
Postgraduate  School  (NPS),  Monterey,  CA  [50].  The  NOTH  design  “combines 
orthogonal  Latin  hypercube  and  uniform  designs  to  create  designs  having  near 
orthogonality  and  excellent  space-filling  properties.”  [50] 

We  used  an  Excel  spreadsheet  developed  by  Professor  Susan  Sanchez  at  the  NPS 
Simulation  Experiments  and  Efficient  Designs  (SEED)  Center  [51]  to  generate  a  NOLH 
design  for  this  thesis.  The  user  specifies  factor  names,  and  maximum  and  minimum 
values  of  factor  levels  as  inputs.  The  spreadsheet  provides  a  NOLH  design  in  the  units  of 
the  problem.  This  worksheet  allows  the  user  to  select  between  different  designs 
depending  on  the  number  of  factors,  with  a  minimum  of  seven  and  maximum  of  29 
factors.  For  other  numbers  of  factors,  NOLHs  can  be  generated  using  the  mixed-integer 
programming  algorithm  of  Hernandez  [52].  We  used  the  spreadsheet  for  up  to  29  factors 
in  this  thesis.  Figure  19  illustrates  a  partial  screenshot  of  the  NOLH  design  for  the  28 
factors  varied  in  this  thesis. 


55 


Figure  19.  Partial  screenshot  of  NOLH  design  spreadsheet  for  29  factor  design, 

after  [51]. 


Sanchez’s  spreadsheet  provides  257  different  combinations  of  factor  values,  using 
the  ranges  of  the  factor  levels.  In  other  words,  it  selects  257  different  design  points  (DP) 
among  the  entire  set  of  input  combinations,  which  provides  an  efficient  design  while 
covering  a  broad  section  of  input  space.  This  property  of  NOLH  designs  is  called  the 
space-filling  property.  We  can  visualize  the  space-filling  property  of  our  NOLH  designs 
by  using  a  scatterplot  matrix.  Figure  20  depicts  a  partial  screenshot  of  the  scatterplot 
matrix  for  the  factors  used  in  our  model.  The  entire  scatterplot  matrix  for  all  factors  is 
included  in  Appendix  A.  The  diagonal  of  the  scatterplot  shows  the  factor  names  while  the 
ranges  are  shown  on  left  side.  The  figure  shows  that  each  pair  of  factors  is  sampled 
throughout  the  possible  region. 
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Figure  20.  Partial  screenshot  of  the  scatterplot  matrix  for  the  factors. 


Another  property  of  our  NOLH  designs  is  near-orthogonality.  Near-orthogonality 
guarantees  the  factors  are  not  confounded.  Orthogonality  of  a  design  can  be  examined 
through  a  color  map  of  correlations,  as  shown  in  Figure  21.  A  red  color  refers  to  a  strong 
positive  correlation,  while  blue  implies  a  strong  negative  correlation  between  the  factors. 
The  grayer  the  diagram  is,  the  nearer  to  orthogonality.  The  diagonal  line  in  the  diagram 


shows  the  correlation  of  a  factor  by  itself  The  correlations  between  all 
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of  factors  are  close  to  zero. 
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Figure  2 1 .  Color  map  on  correlations  for  the  factors  used  in  design. 

C.  MODEL  RUNNING  PROCESS 

Sanchez’s  spreadsheet  provides  257  different  combinations  of  factor  values,  in 
other  words  design  points  (DPs),  using  the  ranges  of  factor  levels.  As  a  result,  we 
obtained  a  DOE  file.  We  entered  the  base  case  MANA  scenario,  in  extensible  Markup 
Language  (XML)  format,  and  the  DOE  file,  in  comma-separated  value  (CSV)  format  into 
a  software  program  called  XStudy,  written  by  SEED  Center  research  associate  Steve 
Upton  [51].  XStudy  enables  the  user  to  map  each  column  in  the  design  file  to  a  specific 
parameter  element  in  MANA,  using  XPaths.  An  XPath  is  a  reference  to  a  specific 
location  in  an  xml  file.  We  also  entered  other  details  about  the  study  design,  such  as  the 
version  of  MANA  and  the  number  of  replications  per  design  point,  into  this  tool,  yielding 
a  single  “study.xml”  file. 

This  study  file  is  used  by  another  program  called  oldmcdata,  also  written  by  Steve 
Upton,  which  programmatically  modifies  the  MANA  XML  file,  producing  a  separate 
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XML  scenario  file  for  eaeh  design  point  [51].  We  used  an  open  souree  software  paekage 
called  Condor,  available  from  the  University  of  Wisconsin  (http://www.cs.wisc.edu/ 
condor),  to  distribute  and  manage  the  MANA  jobs  in  parallel  across  a  set  of  available 
proeessors.  The  oldmcdata  software  ereates  the  set  of  submit.dat  files  needed  by  eondor, 
one  for  eaeh  design  point  job.  A  job  consists  of  a  set  of  replieations  for  one  design  point 
excursion. 

Upon  completion  of  the  runs,  oldmedata  ineludes  a  data  post-proeessor  that 
eombines  the  MANA  summary  file  output  from  the  individual  design  point  excursions 
into  one  CSV  file,  ready  for  use  with  any  data  analysis  software  package.  This  output  file 
eontains  input  faetor  settings  from  the  DOE,  the  random  number  seed,  and  outputs  for 
each  replication.  The  SEED  Center  high  performanee  computing  cluster  configuration 
used  for  these  runs  was  eomposed  of  128  Windows  processors,  with  2  to  4  GB  RAM  per 
proeessor. 

Our  model  is  a  time-stepped,  agent-based,  stochastie  model.  Because  it  is  a 
stochastic  process,  we  need  to  make  a  number  of  replications  for  each  design  point  to 
eapture  the  desired  preeision  of  the  results  and  see  a  range  of  outcomes.  We  determined 
the  desired  number  of  replieations  per  DP  by  using  equation  (1). 

(1) 

I  (Ao-A')  ) 

In  our  caleulations,  we  use  a  or  value  of  0.05  and  a  value  0.10.  a  and  (5  are 
also  called  the  type  1  and  type  II  error  probability,  respeetively.  We  also  want  to  eapture  a 
praetieal  differenee  of  three  terrorists  killed,  a  is  the  standard  deviation  obtained  from 
the  initial  100  replications  of  the  baseline  scenario.  After  plugging  in  all  required 
information,  we  come  out  with  approximately  50  replications  per  DP.  Figure  22 
illustrates  the  number  of  replieations  required  as  a  function  of  the  desired  praetieal 
difference. 
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Sample  Size  Chart 


Figure  22.  Number  of  replications  required  as  a  function  of  desired  practical 

difference. 

After  multiplying  the  number  of  replications  required  per  DP  (50)  by  the  number 
of  DP  (257),  we  get  12,850  runs  per  scenario.  Finally,  we  multiplied  this  number  by 
number  of  scenarios  (8),  and  we  end  up  with  a  total  of  102,800  simulated  engagements. 
These  runs  would  have  taken  over  600  days  to  complete  if  we  used  a  single  processor 
computer.  However,  we  used  a  128-processor  cluster,  which  took  approximately  15  days 
to  complete  all  of  the  runs.  After  15  days,  we  received  the  output  in  a  CSV  file  format, 
which  included  a  total  of  102,800  rows  of  data  to  be  analyzed. 
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V.  DATA  ANALYSIS 


In  this  chapter,  we  define  the  measures  of  effectiveness  (MOEs)  and  the  analysis 
tool  that  we  used  in  this  thesis.  After  providing  general  summary  statistics  for  our  MOEs, 
we  compare  scenarios  in  detail  using  different  comparison  techniques.  Afterward,  we  fit 
regression  models  to  identify  influential  factors. 

A.  MEASURES  OF  EFFECTIVENESS 

The  U.S.  DOD  defines  a  measure  of  effectiveness  (MOE)  as  “a  criterion  used  to 
assess  changes  in  system  behavior,  capability,  or  operational  environment  that  is  tied  to 
measuring  the  attainment  of  an  end  state,  achievement  of  an  objective,  or  creation  of  an 
effect.”  [22]  The  purpose  of  this  thesis  is  to  analyze  the  effectiveness  of  the  use  of 
UCAVs  against  active  terrorists  in  rough  terrain.  We  also  want  to  explore  the 
effectiveness  of  using  F-16s  and  UCAVs  together.  Therefore,  we  have  three  different 
MOEs  in  this  thesis  to  answer  the  research  questions  and  gain  operational  insights. 

First,  we  measure  the  total  number  of  terrorists  killed  (MOE  1),  in  other  words, 
the  number  of  red  casualties  at  the  end  of  each  run.  As  we  explained  in  Chapter  III,  there 
are  120  terrorists  separated  into  three  groups.  Each  air-to-ground  strike  asset  engages  one 
of  the  terrorist  groups  and  kills  a  portion  of  this  group.  MOE  1  is  necessary  but  not 
sufficient  to  understand  the  effectiveness  of  UCAVs  because  UCAVs  can  carry  smaller 
air-to-ground  weapons  compared  to  F-16s.  We  need  to  define  another  MOE  to  better 
explore  the  effectiveness  of  the  UCAVs. 

Second,  we  measure  the  time  to  complete  the  mission  (MOE  2),  i.e.,  counter  the 
terrorists.  The  time  is  a  very  important  MOE  because,  as  we  mentioned  in  early  chapters, 
terrorists  are  well  adapted  to  rough  terrain,  so  they  can  travel  long  distances  in  relatively 
short  time  periods.  Even  if  they  cannot  reach  their  final  destinations,  they  can  pass 
through  villages  or  other  military  facilities  where  they  can  conduct  attacks  and  kill  many 
people. 

Third,  we  measure  the  probability  that  the  terrorists  (red)  reach  their  goal  (MOE 

3).  That  is,  the  probability  that  at  least  one  of  the  terrorist  groups  attacks  a  blue  battalion. 
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In  some  of  the  runs,  blue  forees  eannot  detect  or  classify  one,  two,  or  all  of  the  terrorist 
groups,  which  means  that  the  undetected  terrorist  groups  attack  a  blue  battalion.  If  any  of 
the  blue  battalions  are  attacked,  it  means  a  mission  failure  for  blue  and  a  mission  success 
for  red.  Therefore,  we  created  a  column  named  red  reached  goal.  If  any  of  the  terrorist 
squads  remain  undetected  until  the  end  of  a  run  of  a  maximum  90,000  time  steps,  we 
enter  a  “1”  in  the  corresponding  cell  for  that  column.  Otherwise,  we  give  a  value  of  “0.” 
This  column  provides  an  estimated  probability  for  each  design  point  in  each  scenario 
when  we  average  the  values  over  the  replications. 

B.  ANALYSIS  TOOL 

The  only  analysis  tool  that  we  use  in  this  thesis  is  IMP,  which  was  first  launched 
in  1989  by  SAS  Company  and  is  available  for  purchase  at  http://www.imp.com  [53].  IMP 
software  helps  the  user  interactively  visualize  and  analyze  the  data.  It  makes  importing 
and  processing  the  data  easy  as  well  as  performs  the  appropriate  analysis  techniques.  IMP 
also  has  drag-and-drop  interface  and  dynamically  linked  graphs,  which  allows  users  to 
discover  more  from  their  data. 

The  user  can  apply  various  statistical  techniques  such  as,  linear  and  nonlinear 
regression,  model  comparisons,  partition  trees,  Gaussian  processing,  and  time  series.  In 
this  thesis,  we  used  IMP  software  1 1.2.0  to  analyze  our  simulation  output. 

C.  INITIAL  DATA  ANALYSIS  BY  SCENARIOS 

In  this  section,  we  provide  the  general  results  obtained  from  the  histograms, 
summary  statistics,  and  box  plots  for  the  MOEs  across  our  eight  scenarios.  The  detailed 
figures  for  the  distributions  of  each  MOE,  including  histograms  and  summary  statistics, 
are  provided  in  Appendix  B.  In  this  section,  we  provide  only  a  table  that  summarizes  all 
of  the  results  (see  Table  8).  The  highest  and  the  lowest  values  for  each  MOE  are 
highlighted  in  Table  8. 

We  also  provide  box  plots  for  the  MOEs  categorized  by  scenarios.  Boxplots  show 
the  distribution  of  the  data  in  one  dimension  and  are  appropriate  tools  for  comparing  the 
distributions  of  continuous  variables  between  categorical  groups.  As  we  can  see  from 
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Figure  23,  in  a  box  plot,  the  median  is  bounded  by  the  25%  and  75%  pereentiles.  The 
most  outlying  observations  are  defined  as  outliers  or  extreme  outliers.  JMP  diseriminates 
between  the  data  and  the  outliers,  but  not  the  extreme  outliers. 


Table  8.  The  mean  and  standard  deviation  of  three  MOEs  for  eight  different 

seenarios. 


MOEl 

MOEl 

MOE3 

Scenario 

Mean 

Std.Dev. 

Mean(sec) 

Std.Dev.(sec) 

Mean 

Std.Dev. 

Baseline 

35.61 

11.67 

58,757 

23,937 

0.18 

0.30 

Altl 

30.68 

8.00 

53,051 

23,132 

0.09 

0.22 

Alt2 

32.31 

8.17 

54,580 

23,201 

0.24 

0.24 

Alt3 

30.95 

8.85 

58,505 

24,032 

0.18 

0.31 

Alt4 

31.90 

8.26 

53,855 

22,549 

0.09 

0.21 

Alts 

32.77 

8.74 

63,290 

22,202 

0.21 

0.31 

Alt6 

33.84 

9.27 

56,616 

22,141 

0.13 

0.26 

Alt? 

32.17 

9.64 

61,178 

24,234 

0.22 

0.33 
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Figure  23.  Key  for  reading  the  boxplots  [54]. 


We  will  make  general  comparisons  and  provide  general  insights  in  this  section, 
but  the  detailed  statistical  comparisons  are  provided  in  the  further  sections. 

1.  MOE  (1):  The  Number  of  Terrorists  Killed 

According  to  the  summary  statistics  in  Table  8  and  the  box  plots  on  the  left  side 
of  Figure  24,  the  number  of  terrorists  killed  is  the  highest  in  the  baseline  scenario.  Recall 
that  in  the  baseline  scenario  we  allocate  three  F-16s  to  the  three  regions  on  the  Turkish- 
Iraqi  border.  The  alternative  scenarios  5,  6,  and  7  have  two  F-16s,  while  alternative 
scenarios  2,  3,  and  4  have  only  one  F-16,  and  alternative  scenario  1  does  not  have  any  F- 
16s.  The  F-16s  can  carry  more  effective  weapons  than  the  UCAVs  in  our  model. 
Therefore,  the  baseline  and  the  alternative  scenarios  5,  6,  and  7  have  a  larger  mean 
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number  of  terrorists  killed  compared  to  the  others.  The  alternative  scenario  1  has  the 
smallest  mean  number  of  red  casualties.  When  we  look  at  the  box  plots  for  the  mean 
number  of  red  casualties  in  Figure  24,  we  see  small  differences  between  all  of  the 
scenarios,  but  we  cannot  discriminate  them  correctly  because  the  means  are  very  close  to 
each  other  and  the  variations  are  large.  In  other  words,  there  may  be  statistical  differences 
between  the  scenarios  but  we  cannot  easily  see  them  by  only  looking  at  the  box  plots. 
Note:  the  variation  is  high  because  each  scenario  is  examined  at  257  different  design 
points. 

These  results  are  not  surprising  because  we  always  use  three  air-to-ground  strike 
assets  in  each  run.  Therefore,  we  expect  more  casualties  when  we  use  more  powerful 
weapons.  However,  these  results  show  that  using  even  three  UCAVs  rather  than  three  F- 
16s  does  not  yield  extremely  small  red  casualties.  Moreover,  the  baseline  scenario  has 
only  three  more  red  casualties  than  Alt2,  in  which  one  F-16  and  two  UCAVs  are  used. 
Looking  at  other  MOEs  provides  the  decision-maker  better  insight. 


Figure  24.  Boxplot  for  mean  number  of  terrorists  killed  (left)  and  mean  time 
spent  to  complete  the  mission  (right)  by  scenario. 


2.  MOE  (2):  The  Time  to  Complete  the  Run 

According  to  Table  8,  alternative  scenario  1,  in  which  three  UCAVs  are  allocated 
to  the  three  regions,  has  the  lowest  mean  time  to  counter  all  three  terrorist  groups.  In 
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other  words,  if  we  use  three  UCAVs  along  the  Turkish-Iraqi  border  rather  than  three  F- 
16s,  we  prevent  the  terrorist  groups  from  attacking  blue  assets  in  a  shorter  time. 
Moreover,  the  scenarios  in  which  we  use  two  UCAVs,  alternative  scenarios  2,  3,  and  4, 
have  shorter  mean  times  than  the  ones  in  which  we  use  one  UCAV,  alternative  scenarios 
5,  6,  and  7.  Alternative  scenarios  6  and  3  are  exceptions  for  this  generalization.  However, 
similar  to  MOE  1,  as  shown  in  the  box  plots  on  the  right  side  of  Figure  24,  the  variations 
are  large  also  for  MOE  2.  Therefore,  we  cannot  say  that  differences  between  the  means 
are  statistically  significant  by  only  looking  at  the  summary  statistics  and  the  box  plots. 

3.  MOE  (3):  The  Probability  that  the  Red  Reaches  its  Goal 

Similar  to  the  time  MOE,  alternative  scenarios  that  consist  of  two  UCAVs  and 
one  F-16  have  a  smaller  probability  that  the  terrorists  reach  their  goal  than  the  ones  that 
have  one  UCAV  and  two  F-16s.  According  to  the  summary  statistics  in  Table  8, 
alternative  scenario  1,  which  has  three  UCAVs,  has  the  smallest  probability  for  red 
reaching  the  goal.  Similar  to  MOE  2,  alternative  scenarios  3  and  6  are  exceptions  for  that 
generalization.  The  baseline  scenario  has  a  higher  probability  than  the  two  and  three 
UCAV  scenarios,  with  the  exception  of  alternative  scenario  3. 


Figure  25.  Boxplot  for  mean  probability  of  red  reaching  goal,  by  scenario. 
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We  can  see  the  differences  between  the  means  and  the  variations  from  the  box 
plots  provided  in  Figure  25.  In  contrast  to  MOEs  1  and  2,  MOE  3  has  larger  differences 
between  the  variances.  Alternative  scenarios  1,  2,  4,  and  6  (i.e.,  using  more  UCAVs) 
seem  more  preferable  according  to  the  box  plots. 

The  summary  statistics  and  the  box  plots  provide  general  information  about  the 
data.  However,  we  need  to  apply  different  comparison  techniques  in  order  to  figure  out 
statistical  differences  in  detail  between  the  scenarios.  Therefore,  we  perform  t-test 
comparisons  of  the  means  for  each  MOE  and  scenario  in  the  following  section. 

D.  DETAILED  T-TEST  COMPARISONS  FOR  THE  MEANS  OF  THE  MOES 

A  t-test  is  one  of  the  hypothesis-testing  techniques  that  we  can  use.  The 
information  in  this  section  comes  from  T.  W.  Lucas’s  statistics  class  notes.  A  hypothesis 
is  “a  statement  or  claim  about  a  population  or  populations,  often  concerning  the 
parameters.”  [55]  In  hypothesis-testing,  there  are  two  hypotheses  of  interest:  the  null 
hypothesis  (H^)  and  the  alternative  hypothesis  (H^).  The  null  hypothesis  is  “the  claim 

that  is  initially  assumed  to  be  true”  [55],  and  the  alternative  hypothesis  is  “an  assertion 
that  is  contrary  to  the  null  hypothesis.  Possible  conclusions  from  hypothesis-testing 
analysis  are  reject  or  fail  to  reject  HqJ"  [55].  Equations  (2)  and  (3)  illustrate  the  null 

and  the  alternative  hypothesis  testing,  respectively,  where  ///  and  are  the  means  of  two 
populations. 

(2) 

(3) 

First,  we  calculate  a  test  statistic  using  a  specific  significance  level  ( a ).  If  the  test 
statistic  falls  in  the  rejection  region,  we  reject  the  null  hypothesis;  otherwise,  we  say  that 
there  is  not  enough  evidence  to  reject,  or  in  other  words,  we  fail  to  reject  the  null 
hypothesis.  A  test  statistic  (see  Equation  [4])  is  “a  function  of  the  sample  data  on  which 
the  decision  is  to  be  based.”  [55]  A  rejection  region  is  “the  set  of  all  statistics  values  for 
which  Hq  will  be  rejected”  and  a  significance  level  is  “the  probability  that  we  will 

wrongly  reject  [55] 
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^x,y=  sample  means 
6'j  ,^2  =  sample  std.  deviations 
n^,n2  =  number  of  observations ^ 


Test  statistic  value :  t 


x-y 


(4) 


n  ”2 

Another  way  to  decide  whether  to  reject  or  fail  to  reject  the  null  hypothesis  is  to 
look  at  the  p-values.  A  p-value  is  “the  probability,  if  were  true,  of  obtaining  a  test 


statistics  value  as  extreme  or  more  extreme  than  what  is  observed.”  [55]  We  reject  if 

the  p-value  is  less  than  the  significance  level  (a).  Figure  26  depicts  the  rejection  regions 
and  the  interpretation  of  a  p-value. 


No  F.vidence 
(Not  Siciii! leant) 

Figure  26.  Rejection  regions  (left)  and  interpretation  of  p-value  (right)  [55]. 

We  use  IMP  to  test  the  hypotheses  for  the  means  of  the  MOEs.  Our  null 
hypothesis  is  that  the  sample  mean  of  a  scenario  for  our  MOEs  is  equal  to  the  sample 
mean  of  the  other  scenarios.  In  other  words,  there  is  no  statistical  difference  between  the 
scenarios  in  terms  of  the  means  of  the  MOEs.  Our  alternative  hypothesis  is  that  the 
scenarios  are  statistically  different.  We  always  use  the  significance  level  ( « )  of  0.05  for 
all  calculations  in  this  thesis,  which  means  if  the  p-value  is  less  than  0.05,  we  reject  the 
null  hypothesis.  Thus,  the  two  compared  scenarios  are  statistically  different.  Otherwise, 
we  fail  to  reject  the  null  hypothesis,  which  means  that  the  compared  scenarios  are  not 
statistically  different.  The  following  sections  make  a  t-test  comparison  of  the  scenarios  in 
terms  of  our  three  MOEs. 
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1.  MOE  (1):  The  Number  of  Terrorists  Killed 

The  mean  number  of  terrorists  killed  is  very  close  in  all  of  the  scenarios,  but  there 
are  both  practical  and  statistical  differences  between  some  of  the  scenarios.  A  ractical 
significance  means  that  the  result  is  important  to  the  decision-maker.  For  instance,  for  an 
air-to-ground  operation  like  we  simulate  in  this  thesis,  the  decision-maker  can  say  that 
killing  one  more  terrorist  is  important,  which  means  the  practical  difference  is  one.  If 
killing  10  more  terrorists  makes  a  difference  for  the  decision-maker,  then  the  practical 
difference  is  10.  However,  “no  matter  how  small  the  effect,  if  we  take  a  big  enough 
sample,  we  can  make  it  statistically  significantly  different.”  [55] 
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^  ^  Comparisons  for  each  pair  using  Student's! 
^  Confidence  Quantile 
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Figure  27.  T-test  for  comparison  of  mean  number  of  terrorists  killed. 


Figure  27  shows  the  t-test  comparison  between  the  scenarios  for  a  significance 
level  of  0.05  in  terms  of  the  mean  number  of  terrorists  killed,  or  red  casualties.  The 
connecting  letters  report  section  from  Figure  27  illustrates  the  means  and  the  general 
comparison  between  the  scenarios.  If  the  compared  levels  (scenarios)  have  different 
letters,  it  means  that  they  are  statistically  different  in  terms  of  the  mean  number  of 
terrorists  killed.  Based  on  that  section  of  the  graph,  the  baseline  scenario  is  different  from 
all  other  scenarios.  In  addition,  the  baseline  scenario  has  the  highest  mean  for  red 
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casualties  while  the  alternative  scenario  1  has  the  smallest.  That  is,  three  F-16s  results  in 
the  most  terrorists  killed,  while  3  three  UCAVs  results  in  the  fewest. 

The  ordered  differences  report  section  provides  more  detailed  information,  such 
as  the  p-values  for  tests  on  all  mean  comparisons.  Circled  p-values  are  less  than  the 
significance  level  of  0.05,  which  means  the  corresponding  scenario  pairs  on  the  left  are 
statistically  different.  In  other  word,  the  scenario  pairs  above  the  horizontal  red  line  are 
statistically  significantly  different. 

As  far  as  a  practical  difference  is  concerned,  which  will  be  determined  by  the 
decision-maker,  we  can  say  the  following.  If  we  assume  that  killing  four  more  terrorists  is 
important  for  the  decision-maker,  then  we  can  say  that  the  baseline  scenario  is  practically 
different  form  alternative  scenarios  1  and  3.  A  more  detailed  comparison  report  for  MOE 
1  is  provided  in  Appendix  C. 

2.  MOE  (2):  The  Time  to  Complete  the  Run 

The  time  to  complete  a  run  is  also  an  important  MOE.  Terrorists  are  adapted  to 
the  rough  terrain  in  Turkish-Iraqi  border,  and  they  can  travel  relatively  long  distances  in  a 
short  period  of  time.  They  can  attack  every  village  or  military  facility  that  they  pass 
through  during  this  time  period  even  if  they  cannot  reach  their  final  destination. 
Therefore,  we  want  to  prevent  them  from  proceeding  in  the  shortest  possible  time. 

Figure  28  provides  information  about  t-test  comparison  of  the  scenarios  in  terms 
of  the  time  to  counter  the  terrorists  for  the  a  level  of  0.05.  The  connecting  letters  section 
points  out  that  there  are  statistical  differences  between  some  of  the  scenarios,  such  as 
alternative  scenarios  5  and  7  and  the  baseline  scenario,  or  between  the  baseline  scenario 
and  alternative  scenarios  1,  2,  4,  and  6.  The  scenario  pairs  above  the  horizontal  red  line 
are  statistically  different  because  the  corresponding  p-values  are  less  than  0.05.  Those 
that  are  above  the  horizontal  blue  line  are  considered  practically  different,  assuming 
5,000  seconds  (approximately  1.5  hours)  makes  an  important  difference  for  the  decision¬ 
maker.  On  the  other  hand,  the  baseline  scenario  and  the  alternative  scenarios  3,  6,  and  7 
are  the  same  in  terms  of  the  mean  time  to  complete  the  run.  The  main  takeaway  is  that 
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more  UCAVs  reduces  the  time  to  counter  the  terrorists.  A  more  detailed  comparison 
report  is  provided  in  Appendix  D. 
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Figure  28.  T-test  for  comparison  of  mean  time  to  complete  run. 


3.  MOE  (3):  The  Probability  that  the  Red  Reaches  Its  Goal 

Figure  29  provides  the  information  about  the  t-test  comparison  for  the  scenarios  in 
terms  of  the  mean  probability  that  terrorists  reach  their  goal.  The  main  goal  of  the 
terrorists  is  to  attack  blue  battalions,  so  we  want  this  probability  to  be  as  small  as 
possible. 
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Figure  29.  T-test  for  comparison  of  mean  probability  that  red  reaches  goal. 


According  to  the  Figure  29,  the  paired  scenarios  that  have  a  p-value  less  than  0.05 
are  statistically  different.  The  baseline  scenario  and  alternative  scenarios  1,  2,  and  4  are 
statistically  different,  while  the  baseline  scenario  and  alternative  scenarios  1  and  4  are 
practically  different  as  well,  if  the  decision-maker  takes  the  threshold  probability  as  0.08. 

On  the  other  hand,  we  can  say  form  Figure  29  that  the  baseline  scenario  and 
alternative  scenarios  3,  5,  6,  and  7  are  the  same  in  terms  of  the  probability  that  the  red 
reaches  the  goal.  The  main  takeaway  here  is  that  using  more  UCAVs  reduces  the 

probability  that  the  terrorists  reach  their  goal.  This  follows  from  having  more  sensors 
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along  the  border  and  a  quieker  response  time.  It  is  important  to  note  that  we  assume  that  a 
terrorist  group  withdraws  after  suffering  easualties.  A  more  detailed  eomparison  report  is 
provided  in  Appendix  E. 

4.  Insights  from  the  Scenario  Comparisons 

We  define  three  different  MOEs  in  order  to  look  at  the  problem  from  different 
perspeetives  and  provide  eomprehensive  insight  to  the  decision-maker.  From  the  result  of 
the  initial  data  analysis  and  the  scenario  comparisons,  we  can  say  that  the  advantages  and 
disadvantages  of  the  scenarios  differ  depending  on  the  MOEs.  The  decision-maker  should 
decide  whether  he/she  wants  to  get  more  red  casualties,  or  eliminate  the  terrorists  as 
quickly  as  possible,  or  decrease  the  probability  of  being  attacked  by  the  terrorists. 

In  general,  F-16s  are  better  at  causing  more  red  casualties,  but  UCAVs  kill 
quicker  and  are  better  at  preventing  terrorists  from  attacking  blue.  The  reason  for  this  is 
that  F-16s  have  more  powerful  weapons,  but  UCAVs  have  more  sophisticated  sensors 
(and  are  near  the  border)  so  they  can  detect,  classify,  and  attack  terrorists  in  a  relatively 
shorter  time.  More  specifically,  the  scenarios  where  we  use  two  F-16s  and  one  UCAV  are 
more  advantageous  for  killing  red  than  the  scenarios  with  one  F-16  and  two  UCAVs.  On 
the  other  hand,  the  scenarios  with  two  UCAVs  are  preferable  in  terms  of  MOEs  2  and  3. 

We  recommend  the  decision-maker  to  use  at  least  two  UCAVs  in  that  area  rather 
than  using  three  F-16s.  We  think  that  decreasing  the  probability  of  red’s  attacking  blue  is 
more  important  than  the  number  of  red  casualties.  We  also  want  to  assign  one  of  the 
UCAVs  to  the  region  two  because  statistical  results  show  that  using  UCAVs  in  region 
two  increases  the  performance  in  terms  of  MOE  2  and  3.  In  other  words,  using  F-I6s  in 
region  two  decreases  the  performance  in  terms  of  MOE  2  and  3.  This  may  be  because  of 
the  terrain  effects  in  region  2,  but  we  need  to  make  further  investigations  to  understand 
the  actual  reason,  which  is  beyond  the  scope  of  this  thesis. 

In  sum,  using  three  UCAVs  or  a  combination  of  two  UCAVs  and  one  F-16  seems 
preferable  for  a  decision-maker.  Now,  we  want  to  know  what  kind  of  characteristics  the 
F-16s  and  the  UCAVs  should  have  in  order  to  obtain  better  results.  For  this  purpose,  we 
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perform  regression  analyses  to  find  significant  factors  that  have  an  influence  on  the 
response. 

E.  LINEAR  REGRESSION  ANALYSIS 

A  regression  analysis  is  used  to  figure  out  the  effects  of,  and  the  relationship 
between,  the  input  factors  and  the  response  variables.  A  regression  fits  a  mathematical 
model  and  tries  to  minimize  the  square  distances  to  the  observed  data.  The  general 
mathematical  equation  for  the  linear  regression  is  shown  in  Equation  (5)  [56]. 

7  =  A + ^  (5) 

In  Equation  (5),  y6'.,i  =  0,1,2, ...,p-l  are  unknown  parameters  and  is  called  the 
intercept  term.  Y is  called  the  response  and  X.,...,Xp_j  are  the  predictors,  s  is  an  error 
term,  often  assumed  to  be  normal  [56]. 

In  the  following  section,  we  examine  the  significantly  influential  input  factors, 
including  two-way  interactions  and  nonlinear  effects,  on  the  mean  number  of  terrorists 
killed  (MOE  1),  the  time  to  complete  the  run  (MOE  2),  and  the  probability  that  red 
reaches  the  goal  (MOE  3).  However,  we  only  explain  the  models  for  the  mean  number  of 
red  casualties  in  detail  supported  with  graphs  and  tables.  The  results  for  other  MOEs  are 
summarized,  and  the  detailed  results  are  provided  in  Appendix  F. 

Two  approaches  were  considered  to  fit  regression  models  to  the  factors  we  varied. 
The  first  approach  was  to  fit  the  model,  grouped  by  both  scenario  and  the  design  point.  In 
this  first  approach,  we  obtained  very  low  values  indicating  a  poor  model  fit.  The 
second  approach  was  to  fit  the  model,  collapsing  across  the  scenarios  for  every  design 
point.  The  second  approach  gave  us  higher  R  values  indicating  better  model  fits  with 
fewer  degrees  of  freedom.  Moreover,  the  most  influential  variables  and  the  order  of 
importance  of  these  variables  were  exactly  the  same  for  both  approaches.  In  addition, 
with  fewer  rows  of  data,  the  graphs  are  easier  to  interpret.  Thus,  we  preferred  to  use  the 
second  approach  that  uses  the  summary  data  file  with  257  rows  of  data. 
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1. 


Main  Effects  Model 


In  our  thesis,  we  use  stepwise  regression  in  JMP  to  fit  the  mean  number  of 
terrorists  killed  using  only  the  input  factors  as  the  candidate  influential  terms  in  the 
model,  also  called  a  main  effects  model.  Main  effects  models  are  easy  to  understand  and 
interpret  because  there  are  no  quadratic  terms  or  interactions  in  the  model.  Figure  30 
illustrates  the  statistical  results  of  the  basic  model. 
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Figure  30.  Results  for  main  effects  model  for  mean  number  of  terrorists  killed. 


The  “Summary  of  Fit”  table  in  Figure  30  provides  information  about  how  much 
the  variance  is  explained  by  the  regression  model,  which  is  represented  by  RSquare  (R^). 

The  “Actual  by  Predicted  Plof  ’  visualizes  how  well  the  model  fits  the  data.  The  “Sorted 
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Parameter  Estimates”  table  provides  information  about  the  faetors  and  the  model.  The 
faetors  used  in  the  model  are  listed  under  the  “Term”  eolumn.  These  terms  correspond  to 
X.  in  equation  (5).  The  “Estimate”  column  includes  the  multipliers  ofX.,  which 

correspond  to  /3^  in  equation  (5)  whereas  the  “Std.Error”  column  shows  the  uncertainties. 

The  “t  Ratio”  and  the  “Prob>|t|”  columns  show  if  the  terms  are  necessary  for  the  model 
and  how  much  influence  they  have.  If  the  “Prob>|t|”  value  or  p-value  is  greater  than  the 
selected  significance  level  of  the  model,  which  is  0.05  in  this  thesis,  then  this  term  is 
significant.  An  example  of  the  interpretation  of  the  model  is  the  following.  A  one  unit 
increase  in  F-16’s  hit  probability  (FlbHitProb)  causes  8.94  unit  increases  in  the  mean 
number  of  terrorists  killed. 

According  to  the  results,  there  are  eight  input  factors  that  have  a  significant  effect 
on  the  mean  number  of  terrorists  killed  for  a  significance  level  of  0.05.  The  most 
significant  terms  in  the  model  are  the  weapon  range  and  the  hit  probability  of  the  F-16s 
and  the  UCAVs,  and  the  sensor  classification  range  of  the  UAVs  and  the  UCAVs.  The 
detection  range  of  the  UAVs  also  appears  to  be  significant  in  the  model. 

The  value  is  0.54,  which  means  54%  of  the  variance  is  explained  in  the  model. 
Determining  a  good  value  of  R  depends  on  the  area  of  application.  For  example,  “in  the 
biological  and  social  sciences,  variables  tend  to  be  more  weakly  correlated  and  there  is  a 
lot  of  noise.  We  would  expect  lower  values  of  R^  in  these  areas.  Some  experience  with 
the  particular  area  is  necessary  to  judge  the  R^  well.”  [57]  In  reality,  there  are  a  lot  of 
uncontrollable  factors  in  air-to-ground  operations,  such  as  weather,  terrain,  temperature, 
moisture,  and  system  malfunctions,  etc.  Therefore,  it  is  impossible  to  predict  the  future 
perfectly  for  this  kind  of  operations.  In  order  to  make  our  model  as  realistic  as  possible, 
we  adjusted  the  range  of  the  factors  in  a  way  that  we  can  include  those  noise  factors  in 
the  model.  Moreover,  we  modeled  the  entire  Turkish-Iraqi  border,  which  is  very  large 
compared  to  the  agents  on  the  ground  or  in  the  air.  Therefore,  there  is  a  lot  of  variation  in 
detection  and  classification  of  terrorists  during  the  simulation  because  of  the  LOS 
calculations  (e.g.,  a  small  move  of  an  agent  can  make  it  invisible  to  the  sensors).  As  a 
result,  lower  values  of  R^  are  expected  in  our  model. 
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R  is  not  the  only  factor  for  deciding  whether  a  model  is  good  or  bad.  We  also 
need  to  check  some  assumptions  using  the  regression  diagnostics,  because  the  estimation 
and  inference  from  the  regression  model  depends  on  these  assumptions  [57],  Those 
assumptions  are: 

1 .  The  error  term  s  has  mean  of  zero. 

2.  The  error  term  s  has  constant  variance. 

3.  The  errors  are  normally  distributed. 

4.  The  errors  are  uncorrelated. 

5.  The  relationship  between  the  response  and  the  regression  variables  is 
correct. 

6.  The  regression  variables  are  independent.  [58] 

These  assumptions  can  be  checked  graphically  or  analytically.  Assumptions  1 
through  4  are  related  to  the  residuals.  Figure  31  shows  the  information  about  the  residuals 
for  the  main  effects  model. 
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Figure  3 1 .  Useful  statistics  and  plots  for  residuals  of  main  effects  model. 


The  error  term  has  the  mean  of  1.145x10’'"^,  which  is  very  close  to  zero.  Constant 
variance  assumption  can  be  checked  from  “Residuals  by  Predicted  Plot.”  We  want  all  of 
the  data  to  spread  out  equally  around  the  dotted  blue  line  in  the  graph.  The  residual  by 
predicted  plot  is  not  perfect  in  our  model.  The  normality  of  the  error  terms  can  be 
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checked  visually  by  looking  at  the  “Normal  Quantile  Plot”  or  analytically  by  conducting 
Shapiro-Wilk  test.  The  normal  quantile  plot  seems  good  except  on  the  edges,  but  we  do 
not  pass  the  Shapiro-Wilk  test  because  the  p-value  is  less  than  0.05.  Therefore,  we 
perform  a  Box-Cox  transformation  on  the  response  variable  to  fix  the  constant  variance 
and  satisfy  the  normality  assumption  for  the  residuals.  The  Box-Cox  transformation 
suggests  a  lambda  value  of  0.2,  which  we  can  also  see  from  the  graph  in  Figure  32. 


Figure  32.  Graphical  illustration  of  the  Box-Cox  transformation  in  the  JMP. 

After  a  Box-Cox  transformation,  we  fit  a  new  model  using  the  new  response 
variable  with  the  power  transformation.  The  influential  factors  on  the  response  variable 
remain  the  same  with  previous  model,  and  the  value  increases  from  0.54  to  0.55  (see 
Figure  35).  As  shown  in  Figure  33,  the  residual  by  predicted  plot  seems  better  than  the 
previous  plot,  which  shows  that  the  error  terms  has  constant  variance.  Moreover,  the 
normal  quantile  plot  is  good,  and  we  also  pass  the  Shapiro-Wilk  test  because  our  p-value 
is  greater  than  the  significance  level  of  0.05,  indicating  the  normality  assumption  is  also 
met.  In  addition,  the  error  term  has  a  mean  value  that  is  even  closer  to  zero  than  the  one 
in  the  previous  model. 
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Figure  33.  Statistics  and  plots  for  residuals  after  Box-Cox  transformation  on 

response  variable. 


After  the  first  three  assumptions  are  met,  we  check  whether  the  errors  are 
uncorrelated.  This  assumption  can  be  checked  visually  by  looking  at  the  residual  by 
predicted  plot.  We  would  like  to  see  no  evident  sequence  of  points  [58],  which  is  the  case 
for  the  plot  in  Figure  33.  We  can  check  the  assumptions  5  and  6  from  the  scatter  plot 
visually  [58].  The  scatter  plot  that  we  provide  in  Appendix  A  satisfies  these  assumptions. 
In  addition,  we  want  to  see  whether  there  is  any  influential  point  by  looking  at  the  Cook’s 
distance  values.  Figure  34  shows  that  all  of  the  Cook’s  distance  values  are  less  than  one, 
indicating  no  influential  values. 
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Figure  34.  Cook’s  distance  values  indicate  that  there  are  no  influential  points. 


79 


The  final  main  effects  model  is  provided  in  Figure  35.  Notice  that  the  new  model 
has  the  same  influential  factors  on  the  response  variable  with  slightly  different  t-ratios 
and  p-values.  The  value  is  also  greater  than  the  first  model. 
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Figure  35.  Results  for  main  affects  model  after  Box-Cox  transformation. 


2.  Main  Effects  Model  for  MOE  2  and  MOE  3 

We  also  fit  the  time  (MOE  2)  and  the  probability  that  red  reaches  the  goal  (MOE 
3)  using  only  the  main  effects.  We  use  linear  regression  for  MOE  2,  but  logistic 
regression  for  MOE  3  since  the  response  is  binary.  The  summary  statistics,  parameter 
estimates,  and  the  receiver  operating  characteristics  (ROC)  curve  for  logistic  regression 
are  provided  in  Appendix  F. 

The  results  of  the  linear  regression  for  MOE  2  shows  that  the  most  significant 
factors  in  the  model  are  the  sensor  classification  range,  the  time  between  detections,  the 
sensor  classification  probability  of  the  UAVs,  and  the  time  between  detections  of  the 
UCAVs.  As  the  sensor  capabilities  of  the  UAVs  and  the  UCAVs  increases,  the  time 
decreases.  Different  than  with  MOE  1,  use  of  the  F-16s  does  not  have  any  influence  on 
the  time  to  complete  the  run.  That  is,  the  sensors  of  the  systems  on  the  border  (i.e.,  UAVs 
and  UCAVs)  are  the  most  important  variables  in  quickly  detecting,  classifying,  and 
countering  the  terrorists. 
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The  results  of  the  logistic  regression  for  MOE  3  show  that  the  most  influential 
factors  are  the  UAV-related  factors,  such  as  UAV  sensor  classification  range  and 
probability.  However,  the  UCAV  and  the  F-16-related  factors  also  have  a  strong  effect  on 
the  response  as  well  as  the  terrorist-related  factors. 

3.  Insights  from  the  Main  Effects  Model 

The  main  effects  models  show  that  all  of  the  air  assets  that  we  use  in  the  model 
have  a  strong  influence  on  MOE  1  and  MOE  3  in  some  way.  However,  the  F- 16  does  not 
have  a  significant  influence  on  the  time  (MOE  2).  The  most  influential  factors  are  related 
to  the  weapons  and  the  sensor  capabilities  of  the  air  assets.  Other  factors,  such  as  speed 
and  scramble  time,  appear  to  be  insignificant  for  MOE  1.  Moreover,  there  are  no 
uncontrollable  factors  that  affect  the  response  in  the  main  effects  model  for  MOE  1  and 
MOE  2.  In  addition,  we  do  not  see  any  direct  sensor-related  factors  of  the  F-16s  in  the 
model  for  MOE  1  and  MOE  2,  but  hit  probability  can  be  considered  as  sensor  related  in 
reality  (e.g.,  as  sensor  capability  increases,  the  hit  probability  also  increases).  As  a  result, 
the  decision-makers  should  focus  on  increasing  the  sensor  capabilities  of  the  UAVs  and 
the  UCAVs.  They  also  should  use  more  powerful  weapons  for  the  operations  in  that  area. 

4.  Second  Order  Model 

In  addition  to  the  main  effects  model,  we  also  consider  the  effects  of  the 
interactions  between  the  main  input  factors  as  well  as  the  quadratic  terms.  Using  stepwise 
regression  in  JMP,  we  started  to  construct  our  model  by  adding  the  quadratic  terms  and 
the  interactions.  In  order  to  keep  the  model  as  simple  as  possible,  we  excluded  the  terms 
that  make  no  sense  for  the  model  and  the  terms  that  have  very  small  influence  on  the 
value.  In  addition,  if  a  main  effect  is  included  in  a  quadratic  or  interaction  term,  we  keep 
this  factor  in  the  model  to  achieve  the  model  hierarchy  even  if  it  does  not  have  a 
significant  effect  for  the  model.  The  summary  statistics,  actual  by  predicted  plot,  and 
sorted  parameter  estimates  for  the  second  order  model  is  depicted  in  Figure  36. 

The  second  order  model  has  an  value  of  0.68,  which  means  the  significant 
terms  explain  roughly  68%  of  the  variability  in  the  mean  number  of  red  causalities. 
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Figure  36.  Results  for  main  effects  model  for  mean  number  of  terrorists  killed. 


Notice  that  the  primary  factors  that  exist  in  the  main  effects  model  also  appear  to 
be  the  most  significant  factors  in  the  second  order  model.  Besides  those  factors,  the  time 
between  detection  for  the  UAV  and  the  UCAV  are  significant  in  the  new  model. 
Moreover,  terrorist  classification  range  also  has  a  significant  influence  on  the  number  of 
terrorists  killed.  Specifically,  as  their  classification  range  increases,  so  does  their 
survivability.  There  is  no  quadratic  term  in  the  second  order  model,  but  some  interactions 
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between  the  controllable  factors  or  the  controllable  and  uncontrollable  factors  are  in  the 
model. 


5.  Second  Order  Model  for  MOE  2 

We  also  fit  the  time  to  complete  the  run  using  the  main  effects  and  the  second 
order  terms.  The  summary  statistics  and  the  parameter  estimates  are  provided  in 
Appendix  F. 

Similar  to  the  main  effects  model,  the  results  of  the  second  order  model  for  the 
MOE  2  show  that  the  most  significant  factors  in  the  model  are  the  sensor  classification 
range,  the  time  between  detections,  the  sensor  classification  probability  of  the  UAVs  and 
the  time  between  detections  of  the  UCAVs.  As  the  sensor  capabilities  of  the  UAVs  and 
the  UCAVs  increases,  the  time  decreases.  Different  then  the  main  effects  model,  use  of 
the  F-16s  also  has  an  influence  on  the  response.  The  interaction  between  the  F-16’s  time 
between  detections  and  the  sensor  classification  range  of  partisans  are  in  the  model. 

6.  Insights  from  the  Second  Order  Model 

Similar  to  the  main  effects  models,  in  the  second  order  models  the  most 
significant  factors  are  related  to  the  weapon  and  the  sensor  capabilities  of  the  air  assets. 
Different  from  the  main  effects  models,  the  second  order  models  contain  also  factors 
related  to  the  red  agents,  such  as  sensor  capabilities  of  terrorists.  Also,  other 
uncontrollable  factors,  such  as  communication  capability  of  terrorists  and  partisans,  are  in 
the  model.  Therefore,  preventing  terrorists  from  communicating  with  civil  supporters  in 
the  cities  may  increase  the  number  of  red  casualties.  UCAV  speed,  which  does  not  exist 
in  the  main  effects  model,  is  also  influential  on  the  response.  According  to  the  results,  the 
UCAVs  should  fly  slower  to  increase  the  number  of  red  casualties. 

F.  REGRESSION  TREE  ANALYSIS 

Regression  trees,  also  known  as  partition  trees,  are  another  way  to  analyze  the 
data  and  explore  the  influences  of  the  factors  on  the  response  variable.  The  partition  tree 
is  easy  to  understand  and  interpret  because  of  its  tree  structure.  The  recursive  partitioning 
regression  algorithm  is  the  following. 
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First,  we  partition  a  predictor  by  selecting  a  point  within  the  range  of  this 
predictor  to  decide  from  where  to  split.  Second,  we  calculate  the  mean  of  the  response 
variable  in  that  partition.  We  apply  the  same  procedure  for  each  partition.  Then,  we  fit  a 
model  for  each  partition  and  choose  the  one  that  minimizes  the  residual  sum  of  squares 
(RSS).  Finally,  we  continue  subpartitioning  the  partitions  in  a  recursive  manner  [56].  The 
overall  model  after  each  split  is  shown  in  a  tree  structure. 

We  perform  a  partition  tree  analysis  for  the  mean  number  of  terrorists  killed  using 
JMP.  Figure  37  illustrates  the  first  six  splits  of  the  tree  model.  We  read  a  partition  tree 
from  top  to  bottom.  Each  box  contains  the  name  of  the  factor  and  the  value  for  optimal 
split.  The  mean  and  the  standard  deviation  of  the  number  of  terrorists  killed  are  also 
included  in  each  box  as  well  as  the  number  of  observations  for  that  partition. 

The  results  of  the  partition  tree  analysis  support  the  linear  regression  results  since 
the  most  significant  factors  are  the  same  for  both.  The  F-16  weapon  range  of  6047  meters 
is  the  first  split  in  the  tree.  The  second  and  the  third  splits  are  the  UCAV  weapon  range  of 
4180  and  the  UAV  sensor  classification  range  of  6547,  respectively.  Notice  that  these 
factors  are  also  the  most  significant  factors  in  both  the  main  effects  and  the  second  order 
linear  regression  model.  We  prefer  the  branches  on  the  right  because  we  want  to  have 
higher  red  casualties.  The  red  circle  shows  the  most  preferable  configuration  for  first  six 
splits  where  F-16s  have  a  weapon  range  greater  than  6047  meters  and  the  UCAVs  have  a 
weapon  range  greater  than  4180  meters.  Note:  recall  that  our  weapon  ranges  are  inflated 
since  we  spread  our  terrorists  more  than  they  would  in  the  real  world. 
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Figure  37.  First  six  splits  of  partition  tree  model  for  number  of  terrorists  killed. 


Although  the  value  increases  as  we  continue  splitting,  we  stop  on  the  20th  split 
and  get  an  value  of  0.62.  We  provide  the  tree  model  for  the  first  15  splits  in  Appendix 
G.  Figure  38  displays  the  improvement  of  R^  value  with  number  of  splits  and  the 
contributions  of  the  factors  to  the  explanatory  power  of  the  tree  model. 
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Figure  38.  Improvement  of  R^  value  with  number  of  splits  (right)  and 
contributions  of  factors  to  explanatory  power  of  tree  model  (left). 


The  partition  tree  results  show  a  strong  correlation  with  the  main  effects  and  the 
second  order  linear  regression  results.  The  sensor-  and  weapon-related  factors  are  mostly 
influential  factors.  As  we  continue  splitting,  other  terms  related  to  the  blue  or  red  forces 
are  added  to  the  tree  with  a  small  effect.  Overall,  having  more  powerful  weapons  and 
more  sophisticated  sensors  increases  the  number  of  red  casualties. 
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VI.  CONCLUSION 


A.  OVERVIEW 

We  analyze  the  effectiveness  of  the  use  of  UCAVs  in  conjunction  with  the 
manned  aircraft  to  counter  active  terrorists  in  rough  terrain  in  this  thesis.  We  try  to  figure 
out  whether  replacing  or  combining  the  manned  aircraft  with  UCAVs  is  effective  for  air- 
to-ground  operations  in  a  rough  terrain.  For  this  purpose,  we  build  a  simulation  model 
and  run  102,800  simulated  air-to-ground  attacks  comprising  eight  different  scenarios  of 
manned  aircraft  and  UAV  combinations.  The  results  are  based  on  our  assumption  that 
three  air-to-ground  strike  assets  will  always  be  used.  Therefore,  it  is  impossible  to  cover 
every  aspect  of  these  kinds  of  operations,  but  we  provide  significant  insights  to  the 
decision-maker.  After  performing  data  analysis  techniques  on  the  output,  we  explore 
useful  results  for  the  effectiveness  of  UCAVs  as  well  as  the  significant  factors  that  have  a 
strong  influence  on  the  outcome. 

B.  RESEARCH  QUESTIONS 

The  results  of  this  research  provide  answers  to  the  following  research  questions. 

(1)  How  might  UCAVs  enhance  Turkey’s  ability  to  secure  its  border 
characterized  by  rough  geographical  conditions? 

In  general,  the  results  of  this  research  show  that  using  the  UCAVs  enhances 
Turkey’s  ability  to  secure  the  southeastern  border.  The  scenario  comparisons  point  out 
that  the  scenarios  in  which  we  use  more  UCAVs,  are  statistically  different  in  favor  of 
UCAVs  in  terms  of  the  time  to  complete  the  mission  and  the  probability  that  the  terrorists 
reach  their  goal.  The  scenarios  where  we  use  more  F-16s  are  preferable  in  terms  of  the 
number  of  terrorists  killed.  However,  some  of  the  scenarios  that  have  two  UCAVs  and 
one  F-16  (Alternatives  2  and  4)  have  slightly  less  mean  red  casualties  than  the  baseline. 
Therefore,  the  UCAVs  may  kill  fewer  terrorist  compared  to  the  current  methods,  but  we 
prevent  them  from  attacking  the  blue  more  quickly  and  with  higher  probability. 
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(2)  What  combination  of  UCAVs  and  UAVs  provides  the  same  or  better 
effectiveness  than  the  combination  of  manned  aircraft  and  UAVs  currently 
in  use? 

The  only  scenario  that  we  use  UCAV  and  UAV  combination  only  is  alternative 
scenario  1,  in  which  we  allocate  three  UCAVs  to  the  Turkish-Iraqi  border.  When  the 
response  variable  is  the  number  of  terrorists  killed,  the  alternative  scenario  1  has  the 
lowest  mean.  However,  when  the  response  variable  is  the  time  to  stop  terrorists  or  the 
probability  of  terrorists’  attacking  the  blue  assets,  alternative  scenario  1  is  preferred  over 
the  baseline.  We  think  that  preventing  terrorists  from  attacking  the  blue  forces  is  more 
important  than  killing  more  terrorists,  so  we  say  that  using  three  UCAVs  and  three  UAVs 
provides  better  effectiveness  than  the  currently  used  baseline. 

(3)  Is  there  a  combination  of  manned  aircraft,  UCAVs,  and  UAVs  that 
provides  the  same  or  better  effectiveness  than  the  manned  aircraft  and 
UAVs  in  use? 

The  scenario  comparisons  and  the  statistics  show  that  using  more  F-16s  than 
UCAVs  yields  more  casualties  but  a  higher  probability  of  the  terrorists’  attacking  blue. 
The  preferred  combination  of  the  F-16s  and  the  UCAVs  depends  on  the  decision-maker’s 
measure  of  effectiveness  selection.  Assuming  that  the  decision-maker  always  wants  to 
keep  the  blue  forces  safe,  alternative  scenarios  2  and  4  appear  to  have  better  effectiveness 
than  the  manned  aircraft  and  UAVs  in  use. 

(4)  What  are  the  advantages  and  disadvantages  of  using  UCAVs  and  UAVs 
only,  manned  aircraft  and  UAVs  only,  or  UCAVs,  manned  aircraft,  and 
UAVs? 

The  first  advantage  of  using  UCAVs  and  UAVs  only  (Alternative  1)  is  that  we 
stop  the  terrorists  in  shortest  time  and  with  the  highest  probability.  Therefore,  we  keep 
the  terrorists  away  from  the  villages  and  the  military  facilities,  resulting  less  blue 
casualties.  Parallel  to  the  time,  the  probability  that  terrorist  attack  blue  forces  also 
decreases.  In  addition,  reducing  the  risk  of  the  pilot’s  life  or  decreasing  the  cost  can  be 
considered  as  other  advantages  but  these  areas  are  beyond  the  scope  of  this  thesis.  The 
disadvantage  of  this  configuration  is  having  less  red  casualties. 
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The  advantage  of  using  manned  aircraft  and  UAVs  only  is  having  more  red 
casualties.  On  the  other  hand,  this  combination  yields  more  time  and  the  likelihood  of 
being  attacked  by  the  terrorists. 

The  manned  aircraft,  UCAVs,  and  UAVs  combination  is  advantageous  in  terms 
of  resulting  in  relatively  more  red  casualties  in  general  than  the  UAVs  and  UCAVs 
combination.  However,  it  is  disadvantageous  in  terms  of  giving  terrorists  more  chance  to 
attack  blue. 

C.  OTHER  FINDINGS 

•  The  most  common  conclusion  about  the  factors  is  that  weapon-related  factors  have 
a  strong  influence  on  the  number  of  terrorists  killed  while  the  sensor-related 
factors  are  the  most  effective  factors  on  the  time  of  completion  and  the  probability 
that  terrorists  attack  blue. 

•  The  weapon  range  of  an  F- 16  and  a  UCAV  are  the  two  most  influential  factors  on 
the  number  of  terrorists  killed.  The  main  effects  model,  the  second  order  model, 
and  the  first  two  splits  of  the  partition  tree  analysis  yields  the  F- 16  and  the  UCAV 
weapon  ranges  as  the  most  significant  factors.  Therefore,  we  should  improve 
current  air-to-ground  weapons  if  we  want  to  get  higher  red  casualties. 

•  The  sensor  classification  range  of  the  UAVs  have  a  strong  effect  on  the  number  of 
terrorists  killed,  the  time  to  complete  the  runs,  and  the  probability  that  the  terrorists 
attack  the  blue.  When  we  increase  the  classification  range,  the  number  of  terrorists 
killed  also  increases  while  the  time  and  the  probability  of  being  attacked  by  red 
decreases.  Mounting  more  sophisticated  sensors  on  the  UAVs  should  be 
considered  to  increase  their  effectiveness. 

•  The  hit  probabilities  of  an  F-16  and  of  a  UCAV  are  also  two  of  the  most 
significant  factors  in  all  regression  model  fits.  As  the  hit  probabilities  of  these 
assets  increases,  the  number  of  red  casualties  increases.  Hit  probability  is  mostly 
related  to  the  sensors  in  real  world.  Thus,  increasing  the  sensor  capabilities  also 
increases  the  probability  of  hit.  However,  this  relationship  was  not  considered  in 
this  thesis. 

•  According  to  the  main  effects  and  the  second  order  models,  the  UAV  sensor 
classification  probability  also  has  a  strong  positive  influence  on  the  number  of 
terrorists  killed  and  a  strong  negative  influence  on  the  time  to  complete  the  run. 

•  Another  important  factor  that  increases  the  number  of  terrorists  killed  is  the  sensor 
classification  range  of  the  UCAVs.  Both  the  main  effects  and  the  second  order 
regression  analysis  yield  this  result.  Similar  to  the  UAVs,  the  UCAVs  should  be 
also  configured  with  advanced  sensors  that  can  see  farther. 


89 


•  The  factors  related  to  the  F-16s  do  not  exist  in  the  model  for  the  time  to 
completion  except  in  an  interaction  with  the  classification  of  the  partisan  agents. 
The  sensor-related  factors  of  the  UAVs  and  the  UCAVs  are  the  ones  that  have  a 
significant  influence  on  the  time  to  completion.  We  should  focus  on  the  UAVs  to 
find  and  kill  terrorists  in  a  shorter  time. 

•  The  sensor-related  factors  of  the  UAVs  and  the  UCAVs,  and  the  weapon-related 
factors  of  the  UCAVs,  are  the  most  influential  variables  on  the  probability  of 
terrorists’  attacking  blue.  Similar  to  the  time  to  completion,  we  need  advanced 
unmanned  vehicles  more  than  the  manned  aircraft  to  decrease  the  probability  of 
being  attacked. 

•  The  classification  range  of  the  terrorists  appears  to  be  a  significant  factor  in  the 
second  order  model  fit  on  the  time  to  complete  the  run.  Flying  ata  higher  altitude 
and/or  having  more  silent  aircraft  can  be  considered  to  reduce  this  effect. 

•  Finally,  the  detection  ranges  of  the  terrorists  and  the  partisans  have  a  significant 
influence  on  the  probability  that  terrorists  attack  blue.  Thus,  preventing  partisans 
from  communicating  with  terrorists  can  be  considered. 

D.  FUTURE  RESEARCH 

This  model  is  well  suited  for  several  excursions  that  were  not  done  for  lack  of 
time.  The  followings  are  examples  of  the  many  potential  scenarios,  which  could  be 
covered. 

•  One  aspect  that  could  be  modeled  is  the  impact  of  the  civilians  and  the 
friendly  forces  in  an  urban  area. 

•  The  effect  of  false-positive  detections  should  be  looked  at. 

•  Another  excursion  that  could  be  modeled  is  dedicated  UCAV  sorties  without 
the  UAVs  and  the  manned  aircraft  to  ascertain  a  nearly  optimal  number  of 
UCAVs  required  for  securing  the  entire  border. 

•  Based  on  the  restrictive  rules  of  engagement,  this  research  did  not  explore  an 
offensive  scenario.  Therefore,  modeling  an  offensive  air-to-ground  attack 
scenario  might  better  inform  a  decision-maker  if  conditions  were  to  escalate. 
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APPENDIX  A.  SCATTERPLOT  MATRIX  FOR  ALL  FACTORS 


The  scatterplot  matrix  for  all  factors,  including  controllable  and  uncontrollable 
factor  shows  the  space-filling  property  of  our  nearly  orthogonal  Latin  hypercube  design 
(NOLH). 
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APPENDIX  B.  HISTOGRAMS  AND  SUMMARY  STATISTICS 


The  following  two  figures  show  the  distribution  of  the  number  of  terrorists  killed 
(MOE)  for  all  of  the  seenarios. 
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The  following  two  figures  show  the  distribution  of  the  time  to  eomplete  the  runs 
(MOE  2)  for  all  of  the  scenarios. 
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The  following  two  figures  show  the  distribution  of  the  probability  that  the  red 
reaches  the  goal  (MOE  3)  for  all  of  the  scenarios. 


^  Distributions 
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APPENDIX  C.  DETAILED  COMPARISON  REPORT  FOR  MOE  I 


The  following  figures  provide  the  paired  t-test  eomparisons  of  the  seenarios  in 
terms  of  the  number  of  red  casualties. 
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^  Detailed  Comparisons  Report 
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^  Comparing  Alt6  with  Alt3 
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Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Corrfidence 


^  Comparing  Alt7  with  Altl 


-4-3-2-101234 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


14963  t  Ratio 
0^5  DF 
3j0779  Prob  >  |t| 
-0.0854  Prob>t 
055  Prob  <  t 


1.855278 

2048 

0.0637 

0.0319’ 

0.9681 


^  Comparing  Alt?  with  Alt2 

Difference 

-01406  t  Ratio 

•017436 

Std  Err  Dif 

05065  DF 

2048 

Upper  CL  Dif 

1.4410  Prob  >  tt| 

05616 

Lower  CL  Dif 

-1.7223  Prob  >  t 

03692 

Confidence 

0.95  Prob  <  t 

0.4308 

^  Comparing  Alt?  with  Alt3 

Difference 

13257  t  Ratio 

1319771 

Std  Err  Dif 

05065  DF 

2048 

Upper  CL  Dif 

25073  Prob  >  |t| 

01287 

Lower  CL  Dif 

-03559  Prob  >  t 

0.0644 

Confidence 

055  Prob  <  t 

0.9356 

^  Comparing  Alt?  with  Alt4 

Difference 

03730  t  Ratio 

0338499 

Std  Err  Dif 

05065  DF 

2048 

Upper  CL  Dif 

15546  Prob  >  tt| 

0.7350 

Lower  CL  Dif 

-13086  Prob  >  t 

0.3675 

Confidence 

055  Prob  <  t 

0.6325 

^  Comparing  Ait?  with  AltS 

Difference 

-03975  t  Ratio 

•0.74068 

Std  Err  Dif 

05065  DF 

2048 

Upper  CL  Dif 

05841  Prob  >  tt| 

0.4589 

Lower  CL  Dif 

-21791  Prob  >  t 

0.7706 

Confidence 

0.95  Prob  <  t 

0.2294 

^  Comparing  Alt?  with  Alt6 

Difference 

-15707  t  Ratio 

-2.07162 

Std  Err  Dif 

05065  DF 

2048 

Upper  CL  Dif 

-00891  Prob  >  |t| 

0.0384* 

Lower  CL  Dif 

-33524  Prob  >  t 

0.9808 

Confidence 

0.95  Prob  <  t 

0.0192* 

/■’ 

\ 

\ 
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APPENDIX  D.  DETAILED  COMPARISON  REPORT  FOR  MOE  2 


The  following  figures  provide 


the  paired  t-test  eomparisons  of  the  seenarios  in 


terms  of  the  time  to  complete  the  run. 


^  Detailed  Comparisons  Report 


^  Detailed  Comparisons  Report 


^  Comparing  Altl  with  .Baseline 


Difference 

-57053  t  Ratio 

-2.7885 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

-16923  Prob  >  |tj 

0.0053- 

Lower  CL  Dif 

-9717.7  Prob  >  t 

0S973 

Confidence 

OSS  Prob  <  t 

0,0027' 

’  Comparing  Alt2  with  .Baseline 

Difference 

-41773  t  Ratio 

-2.04196 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

-165.4  Prob  >  |tl 

0,0413' 

Lower  CL  Dif 

-8190.3  Prob  >  t 

09794 

Confidence 

OSS  Prob  <  t 

0,0206' 

’  Comparing  Alt2  with  Altl 

Difference 

1527.4  t  Ratio 

0.74654 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

5539S  Prob  >  |t| 

0.4554 

Lower  CL  Dif 

-2485.0  Prob  >  t 

03277 

Confidence 

OSS  Prob  <  t 

0.7723 

)  Comparing  Alt3  with  .Baseline 

Difference 

-2523  t  Ratio 

-012342 

Std  Err  Dif 

20463  DF 

2048 

Upper  CL  Dif 

3759S  Prob  >  |tl 

0.9018 

Lower  CL  Dif 

-4265.0  Prob  >  t 

0.5491 

Confidence 

OSS  Prob  <  t 

0.4509 

'  Comparing  Alt3  with  Altl 

Difference 

5452.73  t  Ratio 

2J665076 

Std  Err  Dif 

2045S9  DF 

2048 

Upper  CL  Dif 

946517  Prob  >  |t| 

0,0078* 

Lower  CL  Dif 

144038  Prob  >  t 

0.0039* 

Confidence 

0S5  Prob  <  t 

0.9961 

'  Comparing  Alt3  with  Alt2 

Difference 

3925.3  t  Ratio 

1S18536 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

79373  Prob  >  tt| 

0.0552 

Lower  CL  Dif 

-87.1  Prob  >  t 

0,0276' 

Confidence 

OSS  Prob  <  t 

0S724 

’  Comparing  Ait4  with  .Baseline 

Difference 

-4901.9  t  Ratio 

-2J9584 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

-889.4  Prob  >  |t| 

0.0167' 

Lower  CL  Dif 

-8914.3  Prob  >  t 

0S917 

Confidence 

OSS  Prob  <  t 

03083' 

’  Comparing  Ait4  with  Altl 

Difference 

803.4  t  Ratio 

0392658 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

4815.8  Prob  >  |tl 

0,6946 

Lower  CL  Dif 

-3209.1  Prob>t 

0.3473 

Confidence 

OSS  Prob  <  t 

0.6527 

'  Comparing  Ait4  with  Alt2 

Difference 

-724.0  t  Ratio 

-035388 

Std  Err  Dif 

2046.0  DF 

2048 

Upper  CL  Dif 

3288.4  Prob  >  [tj 

0.7235 

Lower  CL  Dif 

-4736.5  Prob  >  t 

0.6383 

Confidence 

OSS  Prob  <  t 

0.3617 
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^  Comparing  Ait4  with  Ait3 


Difference 

-4649.4  t  Ratio 

-2.27242 

Std  Err  Dif 

20463  DF 

2048 

Upper  CL  Dif 

-636S  Prob  >  |tj 

0.0232' 

Lower  CL  Dif 

-86613  Prob  >  t 

09884 

Confidence 

OSS  Prob  <  t 

0.0116* 

'  Comparing  AltS  with  .Baseline 

Difference 

4532.45  t  Ratio 

2.21528 

Std  Err  Dif 

2045.99  DF 

2048 

Upper  CL  Dif 

8544.89  Prob  >  |t| 

0.0269* 

Lower  CL  Dif 

52030  Prob  >  t 

0.0134* 

Confidence 

095  Prob  <  t 

0.9866 

'  Comparing  AitS  with  Altl 

Difference 

10237.7  t  Ratio 

5303781 

Std  Err  Dif 

20463  DF 

2048 

Upper  CL  Dif 

142501  Prob  >  |t| 

'  .OGOr 

Lower  CL  Dif 

62253  Prob  >  t 

^.0001  ■ 

Confidence 

095  Prob  <  t 

1.0000 

*  Comparing  AitS  with  Alt2 

Difference 

87103  t  Ratio 

4257241 

Std  Err  Dif 

20463  OF 

2048 

Upper  CL  Dif 

12722.7  Prob  >  |ti 

.ooor 

Lower  CL  Dif 

46973  Prob  >  t 

•ooor 

Confidence 

095  Prob  <  t 

1.0000 

'  Comparing  AitS  with  AitS 

Difference 

4784.97  t  Ratio 

2338705 

Std  Err  Dif 

2045S9  OF 

2048 

Upper  CL  Dif 

8797.42  Prob  >  |tj 

0.0194* 

Lower  CL  Dif 

77233  Prob  >  t 

0.0097* 

Confidence 

095  Prob  <  t 

09903 

*  Comparing  AitS  with  Ait4 

Difference 

94343  t  Ratio 

4311123 

Std  Err  Dif 

20463  OF 

2048 

Upper  CL  Dif 

134463  Prob  >  |t| 

-  .lOOi 

Lower  CL  Dif 

5421S  Prob  >  t 

'.OOOl* 

Confidence 

095  Prob  <  t 

1.0000 

*  Comparing  Alt6  with  .Baseline 

Difference 

-21413  t  Ratio 

-1.04682 

Std  Err  Dif 

20463  OF 

2048 

Upper  CL  Dif 

1870.7  Prob  >  |tj 

02953 

Lower  CL  Dif 

-61542  Prob  >  t 

03523 

Confidence 

095  Prob  <  t 

01477 

'  Comparing  Alt6  with  Altl 

Difference 

35633  t  Ratio 

1.741685 

Std  Err  Dif 

20463  OF 

2048 

Upper  CL  Dif 

7575S  Prob  >  |t| 

0.0817 

Lower  CL  Dif 

-4493  Prob  >  t 

0.0409' 

Confidence 

095  Prob  <  t 

09591 

*  Comparing  Alt6  with  Alt2 

Difference 

20361  t  Ratio 

0995145 

Std  Err  Dif 

20463  OF 

2048 

Upper  CL  Dif 

6048.5  Prob  >  |t| 

0.3198 

Lower  CL  Dif 

-19764  Prob  >  t 

0.1599 

Confidence 

095  Prob  <  t 

03401 
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^  Comparing  Alt6  with  Alt3 


Difference 
Std  ErrDif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


-1889J  t  Ratio  -052339 

2046.0  DF  2048 

2123.2  Prob  >  |t|  0.3559 

-5901.7  Prob  >  t  0.8220 

055  Prob  <  t  0.1780 


•8000  -4000  0  2000  6000 


^  Comparing  Alt6  with  Alt4 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


2760.1  t  Ratio 
2046.0  DF 
67723  Prob  >  (t) 
-1252.3  Prob  >  t 
055  Prob  <  t 


1349027 

2048 

0.1775 

0.0887 

05113 
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^  Comparing  Alt6  with  AltS 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


-6674  t  Ratio  -33621 

2046  DF  2048 

-2662  Prob  >  |t|  05011' 

-10687  Prob  >  t  05994 

055  Prob  <  t  05006' 
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^  Comparing  Alt?  with  _Baseline 


Difference 
Std  Err  Dif 
Upper  CL  Dif 
Lower  CL  Dif 
Confidence 


2420.3  t  Ratio 
2046.0  DF 
6432.8  Prob  >  tt| 
-1592.1  Prob  >  t 
055  Prob  <  t 


U82966 

2048 

03370 

01185 

05815 
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^  Comparing  Alt?  with  Altl 


Difference 

8125.6  t  Ratio 

3571467 

/-I 

Std  ErrDif 

2046.0  DF 

2048 

/ 

\ 

\ 

Upper  CL  Dif 

12138.0  Prob  >  |t| 

<.0001*  / 

\ 

Lower  CL  Dif 

41133  Prob  >  t 

<.0001*  / 

\  1 

Confidence 

055  Prob  <  t 

1.0000  , — . 

,  ^  1 

-10000  -5000 

c 

\ 

5000  10000 

Comparing  Alt?  with  Alt2 


Difference 

65983  t  Ratio 

3324927 

Std  Err  Dif 

20465  DF 

2048 

Upper  CL  Dif 

106106  Prob  >  |t| 

050ir 

Lower  CL  Dif 

2585.7  Prob  >  t 

05006' 

Confidence 

055  Prob  <  t 

05994 

'  Comparing  Alt?  with  AltS 

Difference 

26725  t  Ratio 

1.306391 

Std  ErrDif 

20465  DF 

2048 

Upper  CL  Dif 

66853  Prob  >  |t| 

01916 

Lower  CL  Dif 

-13396  Prob  >  t 

0.0958 

Confidence 

055  Prob  <  t 

05042 

*  Comparing  Alt?  with  Alt4 

Difference 

73223  t  Ratio 

3.578809 

Std  Err  Dif 

20465  DF 

2048 

Upper  CL  Dif 

11334.7  Prob  >  |t| 

05004* 

Lower  CL  Dif 

33095  Prob  >  t 

05002* 

Confidence 

055  Prob  <  t 

05998 

'  Comparing  Alt?  with  AltS 

Difference 

-21121  t  Ratio 

-153231 

Std  Err  Dif 

20465  DF 

2048 

Upper  CL  Dif 

19003  Prob  >  |t| 

03020 

Lower  CL  Dif 

-61246  Prob  >  t 

05490 

Confidence 

055  Prob  <  t 

01510 

*  Comparing  Alt?  with  Alt6 

Difference 

456212  t  Ratio 

2.229782 

Std  ErrDif 

204559  DF 

2048 

Upper  CL  Dif 

857436  Prob  >  |t| 

05259* 

Lower  CL  Dif 

549.67  Prob  >  t 

05129* 

Confidence 

055  Prob  <  t 

05871 
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APPENDIX  E.  DETAILED  COMPARISON  REPORT  FOR  MOE  3 

The  following  figures  provide  the  paired  t-test  eomparisons  of  the  seenarios  in 
terms  of  the  probability  that  red  reaches  the  goal. 


^  Detailed  Comparisons  Report 
^  Comparing  Aitl  with  _Baseline 

Difference  -0.08887  t  Ratio  -3.63857 

StdErrDif  0.02442  DF 

Upper  CL  Dif  -0.04097  Prob  >  |tl 


Lower  CL  Dif  -013677  Prob  >  t 
Confidence  055  Prob  <  t 


2048 

00003* 

0.9999 

0.0001* 


-010  -0.05  OOO  005 


^  Comparing  Alt2  with  _Baseline 

Difference  -006070  t  Ratio  -2.48519 
StdErrDif  002442  DF  2048 

Upper  CL  Dif  -001280  Prob  >  |tl  0.0130* 
Lower  CL  Dif  -010860  Prob  >  t  05935 

Confidence  0.95  Prob  <  t  0.0065" 

^  Comparing  Alt2  with  Altl 

Difference  002817  t  Ratio  1153384 
StdErrDif  002442  DF  2048 

Upper  CL  Dif  007607  Prob  >  |tj  02489 

Lower  CL  Dif  -001973  Prob  >  t  0.1244 

Confidence  055  Prob  <  t  00756 

^  Comparing  Alt3  with  _Baseline 

Difference  000607  t  Ratio  0248519 
StdErrDif  002442  DF  2048 

Upper  CL  Dif  005397  Prob  >  |tj  00038 
Lower  CL  Dif  -004183  Prob  >  t  0.4019 

Confidence  055  Prob  <  t  0.5981 

^  Comparing  Alt3  with  Altl 


-0.08  -0.04  0.00  0.04 


-008  -0.04  0.00  004  0.08 


^  Detailed  Comparisons  Report 

^  Comparing  Alt4  with  Alt3 

Difference  -0.09198  t  Ratio  -3.76602 

Std  Err  Dif  0.02442  DF  2048 

Upper  CL  Dif  -0.04408  Prob  >  |t|  00002* 

Lower  CL  Dif  -013988  Prob  >  t  05999 

Confidence  055  Prob  <  t  ‘3001* 

) 

^  Comparing  AltS  with  .Baseline 

Difference  0.02934  t  Ratio  1.201176 

StdErrDif  002442  OF  2048 

Upper  CL  Dif  0.07724  Prob  >  W  02298 

Lower  CL  Drf  -0.01856  Prob  >  t  01149 

Confidence  055  Prob  <  t  0-8851 

I 

4  Comparing  AltS  with  Altl 

Difference  0118210  t  Ratio  4039751 

StdErrDif  0024425  DF  2048 

Upper  CL  Dif  0166110  Prob  >  |t|  ^  OOOI  * 

LowverCL  Dif  0070310  Prob  >  t  -  0001* 

Confidence  055  Prob  <  t  lOOOO 

^  Comparing  AltS  with  Ait2 


-010  -005  OOO  005  010 


\ 


-0.08  -004  OOO  004 


-010  -005  OOO  0.05  010 


/Tk  Difference 

0590039  t  Ratio 

3.686367 

]  ^  StdErrDif 

0524425  DF 

2048 

1  Upper  CL  Dif 

0137939  Prob  >  |t| 

05002* 

1  Lower  CL  Dif 

,  Confidence 

0542139  Prob  >  t 

055  Prob  <  t 

0.0001* 

05999  - 

•008  -0.04  0.00  0.04 


-0.10  -005  OOO  0.05  010 


^  Comparing  AltS  with  Alt3 


Difference 

0594942  t  Ratio 

3587094 

Difference 

0.02327  t  Ratio 

0.952657 

Std  Err  Dif 

0524425  DF 

2048 

StdErrDif 

0.02442  DF 

2048 

Upper  CL  Dif 

0142842  Prob  >  |tj 

05001* 

Upper  CL  Dif 

0.07117  Prob  >  lt| 

03409  A 

Lower  CL  Dif 

0547042  Prob  >  t 

<.0001* 

1  Lower  CL  Dif 

-0.02463  Prob  >  t 

0-1704  A 

Confidence 

055  Prob  <  t 

0.9999  : - - 

055  Prob  <  t 

05296  — - 

-010  -0.05  OOO  005 


-0.08  -004  OOO  0.04  0.08 


^  Comparing  Alt3  with  Alt2 

Difference  0066770  t  Ratio  2.73371 

StdErrDif  0.024425  DF  2048 

Upper  CL  Dif  0114671  Prob  >  |tj  0.0065 

Lower  CL  Dif  0018870  Prob  >  t  0.0032’ 

Confidence  055  Prob  <  t  05968 

^  Comparing  Alt4  with  .Baseline 

Difference  -008591  t  Ratio  -33175 

StdErrDif  002442  DF  2048 

Upper  CL  Dif  -003801  Prob  >  |tj  0.0004" 

Lower  CL  Dif  -013381  Prob  >  t  0.9998 

Confidence  055  Prob  <  t  0.0002’ 


-0.06  -0.04  0.00  0.04  0.08 


-010  -0.05  0.00  055  010 


^  Comparing  Alt4  with  Altl 

Difference  050296  t  Ratio  0121073 

StdErrDif  052442  OF  2048 

Upper  CL  Dif  055086  Prob  >  |tj  05036 

Lower  CL  Dif  -0.04494  Prob  >  t  0.4518 

Confidence  055  Prob  <  t  0.5482 

^  Comparing  Alt4  with  Alt2 

Difference  -0.02521  t  Ratio  -1.03231 

StdErrDif  0.02442  OF  2048 

Upper  CL  Dif  052269  Prob  >  |tj  0.3020 

Lower  CL  Dif  -057311  Prob  >t  05490 

Confidence  055  Prob  <  t  01510 


Comparing  AltS  with  Alt4 

Difference  0115253  t  Ratio  4.718677 

StdErrDif  0.024425  DF  2048 

Upper  CL  Dif  0163153  Prob  >  |t|  0001  * 

Lower  CL  Dif  0567353  Prob  >  t  0001* 

Confidence  055  Prob  <  t  15000 

-  Comparing  Alt6  with  .Baseline 

Difference  -0.04584  t  Ratio  -157664 

StdErrDif  0.02442  DF  2048 

Upper  CL  Dif  0.00206  Prob  >  |t|  05607 

Lower  CL  Dif  -0.09374  Prob  >  t  05696 

Confidence  055  Prob  <  t  05304* 

^  Comparing  Alt6  with  Altl 

Difference  0.04304  t  Ratio  1.761937 

StdErrDif  0.02442  DF  2048 

Upper  CL  Dif  0.09094  Prob  >  |t|  05782 

Lower  CL  Dif  -0.00487  Prob  >  t  05391* 

Confidence  0.95  Prob  <  t  05609 

^  Comparing  Alt6  with  Alt2 

Difference  0.01486  t  Ratio  0.608553 

StdErrDif  052442  DF  2048 

Upper  CL  Dif  056276  Prob  >  |t|  03429 

Lower  CL  Dif  -0.03304  Prob  >  t  02714 

Confidence  0.95  Prob  <  t  0.7286 


-010  -055  050  0.05  010 


-058  -054  050  054  058 


-058  -054  050  054  058 


-058  -0.04  0.00  0.04 
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^  Comparing  Alt6  with  Alt3 

Difference  -0j05191  t  Ratio  -212516 

Std  Err  Dif  0.02442  DF  2048 

Upper  CL  Dif  -0j00401  Prob>  |t|  0.0337* 

Lower  CL  Dif  -0.09981  Prob  >  t  0.9832 

Confidence  0.95  Prob  <  t  0.0168* 


-0.08  -0.04  OjOO  0j04  0.08 


^  Comparing  Alt6  with  Alt4 

Difference  0.04008  t  Ratio  1£40863 

Std  Err  Dif  0j02442  DF  2048 

Upper  CL  Dif  0j08798  Prob  >  |t|  0.1010 

Lower  CL  Dif  -0.00782  Prob  >  t  0.0505 

Confidence  055  Prob  <  t  0.9495 


/ 

/ 

\ 

/ 

^  Comparing  Alt?  with  Alt2 

Difference  0106770  t  Ratio 

Std  Err  Dif  0.024425  DF 

Upper  CL  Dif  0154671  Prob  >  |tj 

Lower  CL  Dif  0.058870  Prob  >  t 

Confidence  055  Prob  <  t 

4  Comparing  Alt?  with  AK3 

Difference  0.04000  t  Ratio 
Std  Err  Dif  0.02442  DF 
Upper  CL  Dif  0.08790  Prob  >  |t| 
Lower  CL  Dif  -0.00790  Prob  >  t 

Confidence  055  Prob  <  t 


-010  -0.05  050  0j05  010 


1.637677 

2048 

01016 

00508 

05492 


-0.08 

-004 

OOO 

004 

008 

'  Comparing  Alt6  with  AltS 

Comparing  Alt?  with  Alt4 

Difference 

-007518  t  Ratio 

-3.07781 

/I 

r\ 

Difference  0.131984  t  Ratio 

5.403698 

Std  Err  Dif 

002442  DF 

2048 

/ 

\ 

Std  Err  Dif  0024425  DF 

2048 

Upper  CL  Dif 

-002727  Prob  >  |t| 

0.0021  • 

/ 

\ 

Upper  CL  Dif  0.179885  Prob  >  |tj 

^.0001* 

Lower  CL  Dif 

-012308  Prob  >  t 

0.9989  1 

y 

\ 

Lower  CL  Dif  0.084084  Prob  >  t 

-0001* 

Confidence 

055  Prob  <  t 

0.0011*  1 - 1 

10000 

-0.10 

-0.05 

OOO 

005 

010 

-0.08  -004  0.00  0.04  0.08 


-005  0.00  005  010  0.15 


^  Comparing  Alt?  with  .Baseline 

Difference  004607  t  Ratio  1086196 

Std  Err  Dif  002442  DF  2048 

Upper  CL  Dif  009397  Prob  >  (t|  0.0594 

Lower  CL  Dif  -0.00183  Prob  >  t  0.0297* 

Confidence  0.95  Prob  <  t  05703 


^  Comparing  Alt?  with  AltS 

Difference  0.01673  t  Ratio  0.685021 

Std  Err  Dif  0.02442  DF  2048 

Upper  CL  Dif  0.06463  Prob  >  jtl  04934 


Lower  CL  Dif  -0.03117  Prob  >  t 
Confidence  0.95  Prob  <  t 


02467 

0.7533 


-0.08  -004  OOO  004  0.08 


-0.08  -0.04  OOO  0.04  0.08 


’  Comparing  Alt?  with  Altl 

Difference  0.134942  t  Ratio  5524771 

Std  Err  Dif  0024425  DF  2048 

Upper  CL  Dif  0.182842  Prob  >  |t|  <.0001* 

Lower  CL  Dif  0.087042  Prob  >  t  <  .0001* 

Confidence  055  Prob  <  t  1.0000 


^  Comparing  Alt?  with  Ait6 

Difference  0.091907  t  Ratio  3.762835 

Std  Err  Dif  0024425  DF  2048 

Upper  CL  Dif  0139807  Prob  >  |t|  00002* 

Lower  CL  Dif  0044007  Prob  >  t  <  0001* 

Confidence  0.95  Prob  <  t  05999 


/ 

/ 

\ 

\ 

\ 

_ 

V  1 

-005  000  005  010  015 


-OlO  -0.05  OOO  0.05  0.10 


102 


APPENDIX  F.  THE  REGRESSION  MODELS  FOR  MOE  2 

AND  MOE  3 


The  following  figure  depiets  the  main  effects  linear  regression  model  fit  on  the 
time  to  complete  the  run  (MOE  2). 
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The  following  figure  depiets  the  main  effeets  logistie  regression  model  fit  on  the 
probability  that  red  reaehes  the  goal  (MOE  3). 
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The  following  figure  depiets  the  second  order  linear  regression  model  fit  on  the 
time  to  complete  the  run  (MOE  2). 
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APPENDIX  G.  FIFTEEN  SPLITS  OF  THE  PARTITION  TREE  FOR 
THE  MEAN  NUMBER  OF  TERRORISTS  KILLED 
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